OntoNotes: Corpus Cleanup of Mistaken Agreement Using Word Sense Disambiguation

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 1057–1064 Manchester, August 2008 OntoNotes: Corpus Cleanup of Mistaken Agreement Using Word Sense Disambiguation Liang-Chih Yu and Chung-Hsien Wu Dept. of Computer Science and Information Engineering National Cheng Kung University Tainan, Taiwan, R.O.C. {lcyu,chwu}@csie.ncku.edu.tw Eduard Hovy Information Sciences Institute University of Southern California Marina del Rey, CA 90292, USA hovy@isi.edu Abstract Annotated corpora are only useful if their annotations are consistent. Most large-scale annotation efforts take special measures to reconcile inter-annotator disagreement. To date, however, no-one has investigated how to automatically determine exemplars in which the annotators agree but are wrong. In this paper, we use OntoNotes, a large-scale corpus of semantic annotations, including word senses, predicate-argument structure, ontology linking, and coreference. To de- termine the mistaken agreements in word sense annotation, we employ word sense disambiguation (WSD) to select a set of suspicious candidates for human evaluation. Experiments are conducted from three as- pects (precision, cost-effectiveness ratio, and entropy) to examine the performance of WSD. The experimental results show that WSD is most effective on identifying erro- neous annotations for highly-ambiguous words, while a baseline is better for other cases. The two methods can be combined to improve the cleanup process. This procedure allows us to find approximately 2% remain- ing erroneous agreements in the OntoNotes corpus. A similar procedure can be easily defined to check other annotated corpora. 1 Introduction Word sense annotated corpora are useful re- sources for many natural language applications. Various machine learning algorithms can then be trained on these corpora to improve the applica- tions’ effectiveness. Lately, many such corpora have been developed in different languages, in- cluding SemCor (Miller et al., 1993), LDC-DSO (Ng and Lee, 1996), Hinoki (Kasahara et al., 2004), and the sense annotated corpora with the help of Web users (Chklovski and Mihalcea, 2002). The SENSEVAL1 (Kilgarriff and Palmer, 2000; Kilgarriff, 2001; Mihalcea and Edmonds, 2004) and SemEval-20072 evaluations have also created large amounts of sense tagged data for word sense disambiguation (WSD) competitions. The OntoNotes (Pradhan et al., 2007a; Hovy et al., 2006) project has created a multilingual cor- pus of large-scale semantic annotations, includ- ing word senses, predicate-argument structure, ontology linking, and coreference3. In word sense creation, sense creators generate sense definitions by grouping fine-grained sense distinctions ob- tained from WordNet and dictionaries into more coarse-grained senses. There are two reasons for this grouping instead of using WordNet senses directly. First, people have trouble distinguishing many of the WordNet-level distinctions in real text, and make inconsistent choices; thus the use of coarse-grained senses can improve inter- annotator agreement (ITA) (Palmer et al., 2004; 2006). Second, improved ITA enables machines to more accurately learn to perform sense tagging automatically. Sense grouping in OntoNotes has been calibrated to ensure that ITA averages at least 90%. Table 1 shows the OntoNotes sense 1 http://www.senseval.org 2 http://nlp.cs.swarthmore.edu/semeval 3 Year 1 of the OntoNotes corpus has been re- leased by Linguistic Data Consortium (LDC) (http://www.ldc.upenn.edu) in early 2007. The Year 2 …

Content truncated to 500 words