Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels

Proceedings of NAACL-HLT 2013, pages 556–562, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels David Jurgens Department of Computer Science University of California, Los Angeles jurgens@cs.ucla.edu Abstract Word sense disambiguation aims to identify which meaning of a word is present in a given usage. Gathering word sense annotations is a laborious and difficult task. Several methods have been proposed to gather sense annota- tions using large numbers of untrained anno- tators, with mixed results. We propose three new annotation methodologies for gathering word senses where untrained annotators are allowed to use multiple labels and weight the senses. Our findings show that given the ap- propriate annotation task, untrained workers can obtain at least as high agreement as anno- tators in a controlled setting, and in aggregate generate equally as good of a sense labeling. 1 Introduction Word sense annotation is regarded as one of the most difficult annotation tasks (Artstein and Poesio, 2008) and building manually-annotated corpora with high- quality sense labels is often a time- and resource- consuming task. As a result, nearly all sense-tagged corpora in wide-spread use are created using trained annotators (Hovy et al., 2006; Passonneau et al., 2010), which results in a knowledge acquisition bot- tleneck for training systems that require sense labels (Gale et al., 1992). In other NLP areas, this bot- tleneck has been addressed through gathering anno- tations using many untrained workers on platforms such as Amazon Mechanical Turk (MTurk), a task commonly referred to as crowdsourcing. Recently, several works have proposed gathering sense anno- tations using crowdsourcing (Snow et al., 2008; Bie- mann and Nygaard, 2010; Passonneau et al., 2012b; Rumshisky et al., 2012). However, these meth- ods produce sense labels that are different from the commonly used sense inventories such as WordNet (Fellbaum, 1998) or OntoNotes (Hovy et al., 2006). Furthermore, while Passonneau et al. (2012b) did use WordNet sense labels, they found the quality was well below that of trained experts. We revisit the task of crowdsourcing word sense annotations, focusing on two key aspects: (1) the annotation methodology itself, and (2) the restric- tion to single sense assignment. First, the choice in sense inventory plays an important role in gathering high-quality annotations; fine-grained inventories such as WordNet often contain several related senses for polysemous words, which untrained annotators find difficult to correctly apply in a given context (Chugur et al., 2002; McCarthy, 2006; Palmer et al., 2007; Rumshisky and Batiukova, 2008; Brown et al., 2010). However, many agreement studies have restricted annotators to using a single sense, which can significantly lower inter-annotator agree- ment (IAA) in the presence of ambiguous or poly- semous usages; indeed, multiple studies have shown that when allowed, annotators readily assign multi- ple senses to a single usage (V´eronis, 1998; Mur- ray and Green, 2004; Erk et al., 2009; Passonneau et al., 2012b). Therefore, we focus on annotation methodologies that enable workers to use as many labels as they feel …

Content truncated to 500 words