Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1161–1166, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1161 Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets Mor Geva Tel Aviv University, Allen Institute for AI morgeva@mail.tau.ac.il Yoav Goldberg Bar-Ilan University, Allen Institute for AI yoav.goldberg@gmail.com Jonathan Berant Tel Aviv University, Allen Institute for AI joberant@cs.tau.ac.il Abstract Crowdsourcing has been the prevalent paradigm for creating natural language under- standing datasets in recent years. A common crowdsourcing practice is to recruit a small number of high-quality workers, and have them massively generate examples. Having only a few workers generate the majority of examples raises concerns about data diversity, especially when workers freely generate sentences. In this paper, we perform a series of experiments showing these concerns are evident in three recent NLP datasets. We show that model performance improves when training with annotator identifiers as features, and that models are able to recognize the most productive annotators. Moreover, we show that often models do not generalize well to ex- amples from annotators that did not contribute to the training set. Our findings suggest that annotator bias should be monitored during dataset creation, and that test set annotators should be disjoint from training set annotators. 1 Introduction Generating large datasets has become one of the main drivers of progress in natural language un- derstanding (NLU). The prevalent method for creating new datasets is through crowdsourcing, where examples are generated by workers (Zaidan and Callison-Burch, 2011; Richardson et al., 2013; Bowman et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2017). A common recent practice is to choose a small group of workers who produce high-quality annotations, and massively generate examples using these workers. Having only a few workers annotate the major- ity of dataset examples raises concerns about data diversity and the ability of models to generalize, especially when the crowdsourcing task is to gen- erate free text. If an annotator consistently uses language patterns that correlate with the labels, a neural model can pick up on those, which can lead to an over-estimation of model performance. In this paper, we continue recent efforts to un- derstand biases that are introduced during the pro- cess of data creation (Levy et al., 2015; Schwartz et al., 2017; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Aharoni and Goldberg, 2018; Paun et al., 2018). We investigate this form of bias, termed annotator bias, and perform multiple experiments over three recent NLU datasets: MNLI (Williams et al., 2018), OPENBOOKQA. (Mihaylov et al., 2018), and COMMONSENSEQA (Talmor et al., 2019). First, we establish that annotator information improves model performance by supplying anno- tator IDs as part of the input features. Second, we show that models are able to recognize an- notators that generated many examples, illustrat- ing that annotator information is captured …

Content truncated to 500 words