A Crowdsourced Database of Event Sequence Descriptions for the Acquisition of High-quality Script Knowledge

DeScript: A Crowdsourced Corpus for the Acquisition of High-Quality Script Knowledge Lilian D. A. Wanzare, Alessandra Zarcone, Stefan Thater, Manfred Pinkal Universität des Saarlandes Saarland, 66123, Germany wanzare@coli.uni-saarland.de, zarcone@coli.uni-saarland.de, stth@coli.uni-saarland.de, pinkal@coli.uni-saarland.de Abstract Scripts are standardized event sequences describing typical everyday activities, which play an important role in the computational mod- eling of cognitive abilities (in particular for natural language processing). We present a large-scale crowdsourced collection of explicit linguistic descriptions of script-specific event sequences (40 scenarios with 100 sequences each). The corpus is enriched with crowd- sourced alignment annotation on a subset of the event descriptions, to be used in future work as seed data for automatic alignment of event descriptions (for example via clustering). The event descriptions to be aligned were chosen among those expected to have the strongest corrective effect on the clustering algorithm. The alignment annotation was evaluated against a gold standard of expert anno- tators. The resulting database of partially-aligned script-event descriptions provides a sound empirical basis for inducing high-quality script knowledge, as well as for any task involving alignment and paraphrase detection of events. Keywords: scripts, events, crowdsourcing, paraphrase 1. Introduction When carrying out everyday activities, having a conver- sation, watching movies or reading novels or newspapers, we make abundant use of script knowledge, that is, knowl- edge of standardized event sequences describing typical everyday activities, such as baking a cake or eating in a fast food restaurant (Schank and Abelson, 1977; Barr and Feigenbaum, 1981). Script knowledge plays an important role for the computational modeling of cognitive abilities (in particular for natural language processing), but mak- ing this kind of knowledge available for use in modeling is not easy. On the one hand, the manual creation of wide- coverage knowledge bases is infeasible, due to the size and complexity of relevant script knowledge. On the other hand, texts typically refer only to certain steps in a script and leave a large part of this knowledge implicit, relying on the reader’s ability to infer the full script in detail. Thus, extraction of script knowledge from large text corpora (as done by Chambers and Jurafsky (2009)) is difficult and the outcome can be noisy. In this work, we present a large-scale crowdsourced col- lection and annotation of explicit linguistic descriptions of event patterns, to be used for the automatic acquisi- tion of high-quality script knowledge. This work is part of a larger research effort where we seek to provide a solid empirical basis for high-quality script modeling by inducing script structure from crowdsourced descriptions of typical events, and to investigate methods of text-to- script mapping, using naturalistic texts from crowdsourced stories, which describe real-life experiences and instanti- ate the same scripts (Modi et al., 2016). Predecessors of our work are the OMICS and SMILE corpora (Singh et al., 2002; Regneri et al., 2010), containing multiple event- sequence descriptions (ESDs) for specific activity types or scenarios. Figure 1 shows some example ESDs for the BAKING A CAKE scenario. As can be seen from the examples, the linguistic …

Content truncated to 500 words