Detecting Errors in Discontinuous Structural Annotation

Proceedings of the 43rd Annual Meeting of the ACL, pages 322–329, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Detecting Errors in Discontinuous Structural Annotation Markus Dickinson Department of Linguistics The Ohio State University dickinso@ling.osu.edu W. Detmar Meurers Department of Linguistics The Ohio State University dm@ling.osu.edu Abstract Consistency of corpus annotation is an essential property for the many uses of annotated corpora in computational and theoretical linguistics. While some re- search addresses the detection of inconsis- tencies in positional annotation (e.g., part- of-speech) and continuous structural an- notation (e.g., syntactic constituency), no approach has yet been developed for au- tomatically detecting annotation errors in discontinuous structural annotation. This is significant since the annotation of po- tentially discontinuous stretches of ma- terial is increasingly relevant, from tree- banks for free-word order languages to se- mantic and discourse annotation. In this paper we discuss how the variation n-gram error detection approach (Dickin- son and Meurers, 2003a) can be extended to discontinuous structural annotation. We exemplify the approach by showing how it successfully detects errors in the syntactic annotation of the German TIGER corpus (Brants et al., 2002). 1 Introduction Annotated corpora have at least two kinds of uses: firstly, as training material and as “gold standard” testing material for the development of tools in com- putational linguistics, and secondly, as a source of data for theoretical linguists searching for analyti- cally relevant language patterns. Annotation errors and why they are a problem The high quality annotation present in “gold stan- dard” corpora is generally the result of a manual or semi-automatic mark-up process. The annota- tion thus can contain annotation errors from auto- matic (pre-)processes, human post-editing, or hu- man annotation. The presence of errors creates prob- lems for both computational and theoretical linguis- tic uses, from unreliable training and evaluation of natural language processing technology (e.g., van Halteren, 2000; Kvˇetˇon and Oliva, 2002, and the work mentioned below) to low precision and recall of queries for already rare linguistic phenomena. In- vestigating the quality of linguistic annotation and improving it where possible thus is a key issue for the use of annotated corpora in computational and theoretical linguistics. Illustrating the negative impact of annotation er- rors on computational uses of annotated corpora, van Halteren et al. (2001) compare taggers trained and tested on the Wall Street Journal (WSJ, Marcus et al., 1993) and the Lancaster-Oslo-Bergen (LOB, Johansson, 1986) corpora and find that the results for the WSJ perform significantly worse. They report that the lower accuracy figures are caused by incon- sistencies in the WSJ annotation and that 44% of the errors for their best tagging system were caused by “inconsistently handled cases.” Turning from training to evaluation, Padro and Marquez (1998) highlight the fact that the true ac- curacy of a classifier could be much better or worse than reported, depending on the error rate of the cor- pus used for the evaluation. Evaluating two taggers on the WSJ, they find tagging accuracy rates for am- 322 biguous words …

Content truncated to 500 words