Using Derivation Trees for Informative Treebank Inter-Annotator Agreement Evaluation

Proceedings of NAACL-HLT 2013, pages 550–555, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Using Derivation Trees for Informative Treebank Inter-Annotator Agreement Evaluation Seth Kulick and Ann Bies and Justin Mott and Mohamed Maamouri Linguistic Data Consortium University of Pennsylvania {skulick,bies,jmott,maamouri} @ldc.upenn.edu Beatrice Santorini and Anthony Kroch Department of Linguistics University of Pennsylvania {beatrice,kroch} @ling.upenn.edu Abstract This paper discusses the extension of a sys- tem developed for automatic discovery of tree- bank annotation inconsistencies over an entire corpus to the particular case of evaluation of inter-annotator agreement. This system makes for a more informative IAA evaluation than other systems because it pinpoints the incon- sistencies and groups them by their structural types. We evaluate the system on two corpora - (1) a corpus of English web text, and (2) a corpus of Modern British English. 1 Introduction This paper discusses the extension of a system de- veloped for automatic discovery of treebank annota- tion inconsistencies over an entire corpus to the par- ticular case of evaluation of inter-annotator agree- ment (IAA). In IAA, two or more annotators anno- tate the same sentences, and a comparison identi- fies areas in which the annotators might need more training, or the annotation guidelines some refine- ment. Unlike other IAA evaluation systems, this system application results in a precise pinpointing of inconsistencies and the grouping of inconsistencies by their structural types, making for a more infor- mative IAA evaluation. Treebank annotation, consisting of syntactic structure with words as the terminals, is by its na- ture more complex and therefore more prone to error than many other annotation tasks. However, high an- notation consistency is crucial to providing reliable training and testing data for parsers and linguistic research. Error detection is therefore an important area of research, and the importance of work such as Dickinson and Meurers (2003) is that errors and an- notation inconsistencies might be automatically dis- covered, and once discovered, be targeted for subse- quent quality control. A recent approach to this problem (Kulick et al., 2011; Kulick et al., 2012) (which we will call the KBM system) improves upon Dickinson and Meur- ers (2003) by decomposing the full syntactic tree into smaller units, using ideas from Tree Adjoining Grammar (TAG) (Joshi and Schabes, 1997). This al- lows the comparison to be based on small syntactic units instead of string n-grams, improving the detec- tion of inconsistent annotation. The KBM system, like that of Dickinson and Meurers (2003) before it, is based on the notion of comparing identical strings. In the general case, this is a problematic assumption, since annotation in- consistencies are missed because of superficial word differences between strings which one would want to compare.1 However, this limitation is not present for IAA evaluation, since the strings to compare are, by definition, identical.2 The same is also true of parser evaluation, since the parser output and the gold standard are based on the same sentences. We therefore take the logical step of applying the KBM system developed for …

Content truncated to 500 words