Learning with Annotation Noise

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 280–287, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Learning with Annotation Noise Eyal Beigman Olin Business School Washington University in St. Louis beigman@wustl.edu Beata Beigman Klebanov Kellogg School of Management Northwestern University beata@northwestern.edu Abstract It is usually assumed that the kind of noise existing in annotated data is random clas- sification noise. Yet there is evidence that differences between annotators are not always random attention slips but could result from different biases towards the classification categories, at least for the harder-to-decide cases. Under an annota- tion generation model that takes this into account, there is a hazard that some of the training instances are actually hard cases with unreliable annotations. We show that these are relatively unproblematic for an algorithm operating under the 0-1 loss model, whereas for the commonly used voted perceptron algorithm, hard training cases could result in incorrect prediction on the uncontroversial cases at test time. 1 Introduction It is assumed, often tacitly, that the kind of noise existing in human-annotated datasets used in computational linguistics is random classification noise (Kearns, 1993; Angluin and Laird, 1988), resulting from annotator attention slips randomly distributed across instances. For example, Os- borne (2002) evaluates noise tolerance of shallow parsers, with random classification noise taken to be “crudely approximating annotation errors.” It has been shown, both theoretically and empiri- cally, that this type of noise is tolerated well by the commonly used machine learning algorithms (Cohen, 1997; Blum et al., 1996; Osborne, 2002; Reidsma and Carletta, 2008). Yet this might be overly optimistic. Reidsma and op den Akker (2008) show that apparent dif- ferences between annotators are not random slips of attention but rather result from different biases annotators might have towards the classification categories. When training data comes from one annotator and test data from another, the first an- notator’s biases are sometimes systematic enough for a machine learner to pick them up, with detri- mental results for the algorithm’s performance on the test data. A small subset of doubly anno- tated data (for inter-annotator agreement check) and large chunks of singly annotated data (for training algorithms) is not uncommon in compu- tational linguistics datasets; such a setup is prone to problems if annotators are differently biased.1 Annotator bias is consistent with a number of noise models. For example, it could be that an annotator’s bias is exercised on each and every in- stance, making his preferred category likelier for any instance than in another person’s annotations. Another possibility, recently explored by Beigman Klebanov and Beigman (2009), is that some items are really quite clear-cut for an annotator with any bias, belonging squarely within one particular ca- tegory. However, some instances – termed hard cases therein – are harder to decide upon, and this is where various preferences and biases come into play. In a metaphor annotation study reported by Beigman Klebanov et al. (2008), certain markups received overwhelming annotator support …

Content truncated to 500 words