From Detecting Errors to Automatically Correcting Them

From detecting errors to automatically correcting them Markus Dickinson Department of Linguistics Georgetown University mad87@georgetown.edu Abstract Faced with the problem of annotation er- rors in part-of-speech (POS) annotated corpora, we develop a method for auto- matically correcting such errors. Build- ing on top of a successful error detection method, we first try correcting a corpus us- ing two off-the-shelf POS taggers, based on the idea that they enforce consistency; with this, we find some improvement. Af- ter some discussion of the tagging process, we alter the tagging model to better ac- count for problematic tagging distinctions. This modification results in significantly improved performance, reducing the error rate of the corpus. 1 Introduction Annotated corpora serve as training material and as “gold standard” testing material for the devel- opment of tools in computational linguistics, and as a source of data for theoretical linguists search- ing for relevant language patterns. However, they contain annotation errors, and such errors provide unreliable training and evaluation data, as has been previously shown (see ch. 1 of Dickinson (2005) and references therein). Improving the quality of linguistic annotation where possible is thus a key issue for the use of annotated corpora in computa- tional and theoretical linguistics. Research has gone into automatically detect- ing annotation errors for part-of-speech annota- tion (van Halteren, 2000; Kvˇetˇon and Oliva, 2002; Dickinson and Meurers, 2003), yet there has been virtually no work on automatically or semi- automatically correcting such annotation errors.1 1Oliva (2001) specifies hand-written rules to detect and Automatic correction can speed up corpus im- provement efforts and provide new data for NLP technology training on the corpus. Additionally, an investigation into automatic correction forces us to re-evaluate the technology using the corpus, providing new insights into such technology. We propose in this paper to automatically cor- rect part-of-speech (POS) annotation errors in cor- pora, by adapting existing technology for POS dis- ambiguation. We build the correction work on top of a POS error detection phase, described in section 2. In section 3 we discuss how to eval- uate corpus correction work, given that we have no benchmark corpus to compare with. We turn to the actual work of correction in section 4, us- ing two different POS taggers as automatic cor- rectors and using the Wall Street Journal (WSJ) corpus as our data. After more thoroughly investi- gating how problematic tagging distinctions affect the POS disambiguation task, in section 5 we mod- ify the tagging model in order to better account for these distinctions, and we show this to signifi- cantly reduce the error rate of a corpus. It might be objected that automatic correction of annotation errors will cause information to be lost or will make the corpus worse than it was, but the construction of a large corpus generally requires semi-automated methods of annotation, and automatic tools must be used sensibly at every stage in the corpus building process. Automated annotation methods are not perfect, but humans also add errors, from biases and inconsistent judg- ments. …

Content truncated to 500 words