Detecting Errors in Part-of-Speech Annotation

Detecting Errors in Part-of-Speech Annotation Markus Dickinson� Department of Linguistics�W. Detmar Meurers The Ohio State University�Department of Linguistics The Ohio State University dickinso@ling.osu.edu� dm@ling.osu.edu Abstract We propose a new method for detect- ing errors in "gold-standard" part-of- speech annotation. The approach lo- cates errors with high precision based on n-grams occurring in the corpus with multiple taggings. Two further tech- niques, closed-class analysis and finite- state tagging guide patterns, are dis- cussed. The success of the three ap- proaches is illustrated for the Wall Street Journal corpus as part of the Penn Tree- bank. 1 Introduction Part-of-speech (pos) annotated reference corpora, such as the British National Corpus (Leech et al., 1994), the Penn Treebank (Marcus et al., 1993), or the German Negra Treebank (Skut et al., 1997) play an important role for current work in com- putational linguistics. They provide training ma- terial for research on tagging algorithms and they serve as a gold standard for evaluating the perfor- mance of such tools. High quality, pos-annotated text is also relevant as input for syntactic process- ing, for practical applications such as information extraction, and for linguistic research making use of pos-based corpus queries. The gold-standard pos-annotation for such large reference corpora is generally obtained using an automatic tagger to produce a first annotation, followed by human post-editing. While Sinclair (1992) provides some arguments for prioritizing a fully automated analysis, human post-editing has been shown to significantly reduce the num- ber of pos-annotation errors. Brants (2000) dis- cusses that a single human post-editor reduces the 3.3% error rate in the STTS annotation of the Ger- man Negra corpus produced by the TnT tagger to 1.2%. Baker (1997) also reports an improve- ment of around 2% for a similar experiment car- ried out for an English sample originally tagged with 96.95% accuracy by the CLAWS tagger. And Leech (1997) reports that manual post-editing and correction done for the 2-million word core corpus portion of the BNC, the BNC-sampler, reduced the approximate error rate of 1.7% for the automati- cally obtained annotation to less than 0.3%. While the last figure clearly is a remarkable re- sult, van Halteren (2000), working with the writ- ten half of the BNC-sampler, reports that in 13.6% of the cases where his WPDV tagger disagrees with the BNC annotation, the cause is an error in the BNC annotation. 1 Improving the correct- ness of such gold-standard annotation thus is im- portant for obtaining reliable testing material for pos-tagger research, as well as for the other uses of gold-standard annotation mentioned at the be- ginning of this section—a point which becomes even stronger when one considers that the pos- annotation of most reference corpora contain sig- nificantly more errors than the 0.3% figure re- ported for the BNC-sampler. In this paper, we present three methods for auto- matic detection of annotation errors which remain 'The percentage of disagreement caused by BNC errors rises to 20.5% for a tagger trained on the entire corpus. 107 despite human post-editing, and sometimes …

Content truncated to 500 words