Detecting Errors in Automatically-Parsed Dependency Relations

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 729–738, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Detecting Errors in Automatically-Parsed Dependency Relations Markus Dickinson Indiana University md7@indiana.edu Abstract We outline different methods to detect er- rors in automatically-parsed dependency corpora, by comparing so-called depen- dency rules to their representation in the training data and flagging anomalous ones. By comparing each new rule to every rel- evant rule from training, we can identify parts of parse trees which are likely erro- neous. Even the relatively simple methods of comparison we propose show promise for speeding up the annotation process. 1 Introduction and Motivation Given the need for high-quality dependency parses in applications such as statistical machine transla- tion (Xu et al., 2009), natural language generation (Wan et al., 2009), and text summarization evalu- ation (Owczarzak, 2009), there is a corresponding need for high-quality dependency annotation, for the training and evaluation of dependency parsers (Buchholz and Marsi, 2006). Furthermore, pars- ing accuracy degrades unless sufficient amounts of labeled training data from the same domain are available (e.g., Gildea, 2001; Sekine, 1997), and thus we need larger and more varied anno- tated treebanks, covering a wide range of domains. However, there is a bottleneck in obtaining an- notation, due to the need for manual interven- tion in annotating a treebank. One approach is to develop automatically-parsed corpora (van No- ord and Bouma, 2009), but a natural disadvantage with such data is that it contains parsing errors. Identifying the most problematic parses for human post-processing could combine the benefits of au- tomatic and manual annotation, by allowing a hu- man annotator to efficiently correct automatic er- rors. We thus set out in this paper to detect errors in automatically-parsed data. If annotated corpora are to grow in scale and re- tain a high quality, annotation errors which arise from automatic processing must be minimized, as errors have a negative impact on training and eval- uation of NLP technology (see discussion and ref- erences in Boyd et al., 2008, sec. 1). There is work on detecting errors in dependency corpus annota- tion (Boyd et al., 2008), but this is based on finding inconsistencies in annotation for identical recur- ring strings. This emphasis on identical strings can result in high precision, but many strings do not re- cur, negatively impacting the recall of error detec- tion. Furthermore, since the same strings often re- ceive the same automatic parse, the types of incon- sistencies detected are likely to have resulted from manual annotation. While we can build from the insight that simple methods can provide reliable annotation checks, we need an approach which re- lies on more general properties of the dependency structures, in order to develop techniques which work for automatically-parsed corpora. Developing techniques to detect errors in parses in a way which is independent of corpus and parser has fairly broad implications. By using only the information available in a training corpus, the methods we explore are …

Content truncated to 500 words