Crowdsourcing a Large Dataset of Domain-Specific Context-Sensitive Semantic Verb Relations

Crowdsourcing a Large Dataset of Domain-Specific Context-Sensitive Semantic Verb Relations Maria Sukhareva†, Judith Eckle-Kohler†, Ivan Habernal†, Iryna Gurevych†‡ † Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt, Hochschulstr. 10, 64289 Darmstadt, Germany ‡ Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research and Educational Information, Schloßstr. 29, 60486 Frankfurt, Germany http://www.ukp.tu-darmstadt.de Abstract We present a new large dataset of 12403 context-sensitive verb relations manually annotated via crowdsourcing. These relations capture fine-grained semantic information between verb-centric propositions, such as temporal or entailment relations. We propose a novel semantic verb relation scheme and design a multi-step annotation approach for scaling-up the annotations using crowdsourcing. We employ several quality measures and report on agreement scores. The resulting dataset is available under a permissive CreativeCommons license at www.ukp.tu-darmstadt.de/data/verb-relations/. It represents a valuable resource for various applications, such as automatic information consolidation or automatic summarization. Keywords: crowdsourcing, semantic relations, new resource 1. Introduction Automatic information consolidation (Saracevic, 1986) in the sense of supporting users in identifying and assimilat- ing a large set of relevant statements found across multi- tudes of heterogeneous documents has recently gained at- tention (Wurzinger, 2010). Semantic relations between text segments play a key role in helping users to explore and navigate in the consolidated information. Often, the seman- tic relation between two segments is triggered by lexical- semantic relations between word senses in the two text seg- ments. In our work, we consider the particular domain of everyday educational topics. This domain contains rather generic language, i.e., has low coverage of named entities. Consider the following example where the lexical-semantic happens-before relation between the verb senses experience an emotion and report an emotion triggers a happens-before relation between the two sentences (1) and (2). (1) Students with learning and behavioral disabil- ities are more likely to experience school stress. (2) Students with learning and behavioral disabil- ities report school stress as the primary reason for drug use. A system for information consolidation could strongly ben- efit from a resource of lexical semantic relations between verb senses, such as the happens-before relation given above. This kind of information would help the system to reveal an inference between the two sentences (1) and (2) which would otherwise remain hidden, because it is not expressed on the text surface via linguistic means, such as verb tense or discourse connectives. Apart from the tempo- ral happens-before relation between verb senses, there are other relations which are important for information consol- idation, e.g., the cause relation. Although WordNet (Fellbaum, 1998) contains semantic re- lations between verb senses, it has a very limited cover- age of semantic verb relations, in particular for domain- specific text (Levy et al., 2014). While various approaches for the automatic acquisition of lexical-semantic relations between verbs from large-scale corpora have been devel- oped (e.g., (Lin and Pantel, 2001; Chklovski and Pantel, 2004; Hashimoto et al., 2009)), the extracted relations are not specified on the sense level, but rather on the word level. For instance, Hashimoto et al. …

Content truncated to 500 words