Some Empirical Evidence for Annotation Noise in a Benchmarked Dataset

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 438–446, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Some Empirical Evidence for Annotation Noise in a Benchmarked Dataset Beata Beigman Klebanov Kellogg School of Management Northwestern University beata@northwestern.edu Eyal Beigman Washington University in St. Louis beigman@wustl.edu Abstract A number of recent articles in computational linguistics venues called for a closer exami- nation of the type of noise present in anno- tated datasets used for benchmarking (Rei- dsma and Carletta, 2008; Beigman Klebanov and Beigman, 2009). In particular, Beigman Klebanov and Beigman articulated a type of noise they call annotation noise and showed that in worst case such noise can severely degrade the generalization ability of a linear classifier (Beigman and Beigman Klebanov, 2009). In this paper, we provide quantita- tive empirical evidence for the existence of this type of noise in a recently benchmarked dataset. The proposed methodology can be used to zero in on unreliable instances, facili- tating generation of cleaner gold standards for benchmarking. 1 Introduction Traditionally, studies in computational linguistics use few trained annotators. Lately this might be changing, as inexpensive annotators are available in large numbers through projects like Amazon Me- chanical Turk or through online games where an- notations are produced as a by-product (Poesio et al., 2008; von Ahn, 2006), and, at least for certain tasks, the quality of multiple non-expert annotations is close to that of a small number of experts (Snow et al., 2008; Callison-Burch, 2009). Apart from the reduced costs, mass annotation is a promising way to get detailed information about the dataset, such as the level of difficulty of the dif- ference instances. Such information is important both from the linguistic and from the machine learn- ing perspective, as the existence of a group of in- stances difficult enough to look like they have been labeled by random guesses can in the worst case induce the machine learner training on the dataset to misclassify a constant proportion of easy, non- controversial instances, as well as produce incor- rect comparative results in a benchmarking setting (Beigman Klebanov and Beigman, 2009; Beigman and Beigman Klebanov, 2009) . In this article, we employ annotation generation models to estimate the types of instances in a multi- ply annotated dataset for a binary classification task. We provide the first quantitative empirical demon- stration, to our knowledge, of the existence of what Beigman Klebanov and Beigman (2009) call “anno- tation noise” in a benchmarked dataset, that is, for a case where instances cannot be plausibly assigned to just two classes, and where instances in the third class can be plausibly described as having been an- notated by flips of a nearly fair coin. The ability to identify such instances helps improve the gold stan- dard by eliminating them, and allows further empiri- cal investigation of their impact on machine learning for the task in question. 2 Generative models of annotation We present a graphical model for the …

Content truncated to 500 words