A Crowdsourced Frame Disambiguation Corpus with Ambiguity

Proceedings of NAACL-HLT 2019, pages 2164–2170 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2164 A Crowdsourced Frame Disambiguation Corpus with Ambiguity Anca Dumitrache FD Mediagroep, Vrije Universiteit Amsterdam Netherlands anca.dmtrch@gmail.com Lora Aroyo Google USA l.m.aroyo@gmail.com Chris Welty Google USA cawelty@gmail.com Abstract We present a resource for the task of FrameNet semantic frame disambiguation of over 5,000 word-sentence pairs from the Wikipedia cor- pus. The annotations were collected using a novel crowdsourcing approach with mul- tiple workers per sentence to capture inter- annotator disagreement. In contrast to the typical approach of attributing the best single frame to each word, we provide a list of frames with disagreement-based scores that express the confidence with which each frame applies to the word. This is based on the idea that inter-annotator disagreement is at least partly caused by ambiguity that is inherent to the text and frames. We have found many examples where the semantics of individual frames over- lap sufficiently to make them acceptable alter- natives for interpreting a sentence. We have argued that ignoring this ambiguity creates an overly arbitrary target for training and eval- uating natural language processing systems - if humans cannot agree, why would we ex- pect the correct answer from a machine to be any different? To process this data we also utilized an expanded lemma-set provided by the Framester system, which merges FN with WordNet to enhance coverage. Our dataset includes annotations of 1,000 sentence-word pairs whose lemmas are not part of FN. Finally we present metrics for evaluating frame disam- biguation systems that account for ambiguity. 1 Introduction Crowdsourcing has been a popular method to col- lect corpora for a variety of natural language pro- cessing tasks (Snow et al., 2008), although one of its downsides is the crowd’s lack of domain knowledge that is helpful in solving some tasks. Semantic frame disambiguation is an example of a complex natural language processing task that is usually performed by linguistic experts, sub- jected to strict annotation guidelines and quality control (Baker, 2012). The theory of frame se- mantics (J Fillmore, 1982) defines a frame as an abstract representation of a word sense, describing a type of entity, relation, or event, together with the associated roles implied by the frame. The FrameNet (FN) corpus (Baker et al., 1998) is a col- lection of semantic frames, together with a corpus of documents annotated with these frames. Simi- larly to word-sense disambiguation, frame disam- biguation is the task of obtaining the correct frame for each word, since many words have multiple possible meanings. Using domain experts for frame disambigua- tion is expensive and time consuming, resulting in small corpora for this task that do not scale well for modern machine learning methods – FN ver- sion 1.7, the latest one at the time of writing, con- tains only about 10,000 sentences annotated with frames. Furthermore, only using one expert to per- form the annotation makes it difficult to capture any diversity of perspectives. There …

Content truncated to 500 words