Noise or additional information? Leveraging crowdsource annotation item agreement for natural language tasks.

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291–297, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Noise or additional information? Leveraging crowdsource annotation item agreement for natural language tasks. Emily K. Jamison ‡ and Iryna Gurevych †‡ ‡Ubiquitous Knowledge Processing Lab (UKP-TUDA), Department of Computer Science, Technische Universit¨at Darmstadt † Ubiquitous Knowledge Processing Lab (UKP-DIPF), German Institute for Educational Research http://www.ukp.tu-darmstadt.de Abstract In order to reduce noise in training data, most natural language crowdsourcing an- notation tasks gather redundant labels and aggregate them into an integrated label, which is provided to the classifier. How- ever, aggregation discards potentially use- ful information from linguistically am- biguous instances. For five natural language tasks, we pass item agreement on to the task classifier via soft labeling and low-agreement filter- ing of the training dataset. We find a sta- tistically significant benefit from low item agreement training filtering in four of our five tasks, and no systematic benefit from soft labeling. 1 Introduction Crowdsourcing is a cheap and increasingly- utilized source of annotation labels. In a typical annotation task, five or ten labels are collected for an instance, and are aggregated together into an integrated label. The high number of labels is used to compensate for worker bias, task misun- derstanding, lack of interest, incompetance, and malicious intent (Wauthier and Jordan, 2011). Majority voting for label aggregation has been found effective in filtering noisy labels (Nowak and R¨uger, 2010). Labels can be aggregated un- der weighted conditions reflecting the reliability of the annotator (Whitehill et al., 2009; Welinder et al., 2010). Certain classifiers are also robust to random (unbiased) label noise (Tibshirani and Manning, 2014; Beigman and Beigman Klebanov, 2009). However, minority label information is dis- carded by aggregation, and when the labels were gathered under controlled circumstances, these la- bels may reflect linguistic intuition and contain useful information (Plank et al., 2014b). Two al- ternative strategies that allow the classifier to learn from the item agreement include training instance filtering and soft labeling. Filtering training in- stances by item agreement removes low agree- ment instances from the training set. Soft labeling assigns a classifier weight to a training instance based on the item agreement. Consider two Affect Recognition instances and their Krippendorff (1970)’s α item agreement : Text: India’s Taj Mahal gets facelift Sadness Rating (0-100): 8.0 α Agreement (-1.0 – 1.0): 0.7 Figure 1: Affect Recognition Easy Case. Text: After Iraq trip, Clinton proposes war limits Sadness Rating (0-100): 12.5 α Agreement (-1.0 – 1.0): -0.1 Figure 2: Affect Recognition Hard Case. In Figure 1, annotators mostly agreed that the headline expresses little sadness. But in Figure 2, the low item agreement may be caused by instance difficulty (i.e., Is a war zone sad or just bad?): a Hard Case (Zeman, 2010). Previous work (Beigman Klebanov and Beigman, 2014; Beigman and Beigman Klebanov, 2009) has shown that training strategy may affect Hard and Easy Case test instances differently. In this work, for five natural …

Content truncated to 500 words