Evaluating the Impact of Coder Errors on Active Learning

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 43–51, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Evaluating the Impact of Coder Errors on Active Learning Ines Rehbein Computational Linguistics Saarland University rehbein@coli.uni-sb.de Josef Ruppenhofer Computational Linguistics Saarland University josefr@coli.uni-sb.de Abstract Active Learning (AL) has been proposed as a technique to reduce the amount of annotated data needed in the context of supervised clas- sification. While various simulation studies for a number of NLP tasks have shown that AL works well on goldstandard data, there is some doubt whether the approach can be suc- cessful when applied to noisy, real-world data sets. This paper presents a thorough evalua- tion of the impact of annotation noise on AL and shows that systematic noise resulting from biased coder decisions can seriously harm the AL process. We present a method to filter out inconsistent annotations during AL and show that this makes AL far more robust when ap- plied to noisy data. 1 Introduction Supervised machine learning techniques are still the mainstay for many NLP tasks. There is, how- ever, a well-known bottleneck for these approaches: the amount of high-quality data needed for train- ing, mostly obtained by human annotation. Active Learning (AL) has been proposed as a promising ap- proach to reduce the amount of time and cost for hu- man annotation. The idea behind active learning is quite intuitive: instead of annotating a large number of randomly picked instances we carefully select a small number of instances that are maximally infor- mative for the machine learning classifier. Thus a smaller set of data points is able to boost classifier performance and to yield an accuracy comparable to the one obtained when training the same system on a larger set of randomly chosen data. Active learning has been applied to several NLP tasks like part-of-speech tagging (Ringger et al., 2007), chunking (Ngai and Yarowsky, 2000), syn- tactic parsing (Osborne and Baldridge, 2004; Hwa, 2004), Named Entity Recognition (Shen et al., 2004; Laws and Sch¨utze, 2008; Tomanek and Hahn, 2009), Word Sense Disambiguation (Chen et al., 2006; Zhu and Hovy, 2007; Chan and Ng, 2007), text classification (Tong and Koller, 1998) or statis- tical machine translation (Haffari and Sarkar, 2009), and has been shown to reduce the amount of anno- tated data needed to achieve a certain classifier per- formance, sometimes by as much as half. Most of these studies, however, have only simulated the ac- tive learning process using goldstandard data. This setting is crucially different from a real world sce- nario where we have to deal with erroneous data and inconsistent annotation decisions made by the human annotators. While simulations are an indis- pensable instrument to test different parameters and settings, it has been shown that when applying AL to highly ambiguous tasks like e.g. Word Sense Disambiguation (WSD) with fine-grained sense dis- tinctions, AL can actually harm the learning process (Dang, 2004; Rehbein et al., 2010). Dang suggests that the lack of a …

Content truncated to 500 words