Detecting annotation noise in automatically labelled data

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1160–1170 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1107 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1160–1170 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1107 Detecting annotation noise in automatically labelled data Ines Rehbein Josef Ruppenhofer IDS Mannheim/University of Heidelberg, Germany Leibniz Science Campus “Empirical Linguistics and Computational Language Modeling” rehbein@cl.uni-heidelberg.de, ruppenhofer@ids-mannheim.de Abstract We introduce a method for error detec- tion in automatically annotated text, aimed at supporting the creation of high-quality language resources at affordable cost. Our method combines an unsupervised gener- ative model with human supervision from active learning. We test our approach on in-domain and out-of-domain data in two languages, in AL simulations and in a real world setting. For all settings, the results show that our method is able to detect annotation errors with high precision and high recall. 1 Introduction Until recently, most of the work in Computational Linguistics has been focussed on standard written text, often from newswire. The emergence of two new research areas, Digital Humanities and Com- putational Sociolinguistics, have however shifted the interest towards large, noisy text collections from various sources. More and more researchers are working with social media text, historical data, or spoken language transcripts, to name but a few. Thus the need for NLP tools that are able to pro- cess this data has become more and more appar- ent, and has triggered a lot of work on domain adaptation and on developing more robust prepro- cessing tools. Studies are usually carried out on large amounts of data, and thus fully manual an- notation or even error correction of automatically prelabelled text is not feasible. Given the impor- tance of identifying noisy annotations in automat- ically annotated data, it is all the more surpris- ing that up to now this area of research has been severely understudied. This paper addresses this gap and presents a method for error detection in automatically la- belled text. As test cases, we use POS tagging and Named Entity Recognition, both standard prepro- cessing steps for many NLP applications. How- ever, our approach is general and can also be ap- plied to other classification tasks. Our approach is based on the work of Hovy et al. (2013) who develop a generative model for es- timating the reliability of multiple annotators in a crowdsourcing setting. We adapt the generative model to the task of finding errors in automatically labelled data by integrating it in an active learning (AL) framework. We first show that the approach of Hovy et al. (2013) on its own is not able to beat a strong baseline. We then present our integrated model, in which we impose human supervision on the generative model through AL, and show that we are able to achieve substantial improvements in two different tasks and for two languages. Our contributions are the following. We …

Content truncated to 500 words