Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1779–1789 May 2-6, 2023 ©2023 Association for Computational Linguistics Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions Mihir Parmar1∗ Swaroop Mishra1∗ Mor Geva2† Chitta Baral1 1Arizona State University 2Allen Institute for AI {mparmar3, srmishr1, chitta}@asu.edu, pipek@google.com Abstract In recent years, progress in NLU has been driven by benchmarks. These benchmarks are typically collected by crowdsourcing, where annotators write examples based on annotation instructions crafted by dataset creators. In this work, we hypothesize that annotators pick up on patterns in the crowdsourcing instructions, which bias them to write many similar exam- ples that are then over-represented in the col- lected data. We study this form of bias, termed instruction bias, in 14 recent NLU benchmarks, showing that instruction examples often ex- hibit concrete patterns, which are propagated by crowdworkers to the collected data. This extends previous work (Geva et al., 2019) and raises a new concern of whether we are mod- eling the dataset creator’s instructions, rather than the task. Through a series of experiments, we show that, indeed, instruction bias can lead to overestimation of model performance, and that models struggle to generalize beyond bi- ases originating in the crowdsourcing instruc- tions. We further analyze the influence of in- struction bias in terms of pattern frequency and model size, and derive concrete recommenda- tions for creating future NLU benchmarks.1 1 Introduction Benchmarks have been proven pivotal for driv- ing progress in Natural Language Understanding (NLU) in recent years (Rogers et al., 2021; Bach et al., 2022; Wang et al., 2022). Nowadays, NLU benchmarks are mostly created through crowd- sourcing, where crowdworkers write examples fol- lowing annotation instructions crafted by dataset creators (Callison-Burch and Dredze, 2010; Zheng et al., 2018; Suhr et al., 2021). The instructions typ- ically include a short description of the task, along ∗Equal Contribution †Now at Google Research 1Code and data is available at https://github.com/ Mihir3009/instruction-bias. with several examples (Dasigi et al., 2019; Zhou et al., 2019; Sakaguchi et al., 2020). Despite the vast success of this method, past stud- ies have shown that data collected through crowd- sourcing often exhibit various biases that lead to overestimation of model performance (Schwartz et al., 2017; Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Le Bras et al., 2020; Mishra et al., 2020a; Mishra and Arunkumar, 2021; Het- tiachchi et al., 2021). Such biases are often at- tributed to annotator-related biases, such as writ- ing style and background knowledge (Gururangan et al., 2018; Geva et al., 2019) (see more discussion on related work in §A). In this work, we propose that biases in crowd- sourced NLU benchmarks often originate at an early stage in the data collection process of de- signing the annotation task. In particular, we hy- pothesize that task instructions provided by dataset creators, which serve as the guiding principles for annotators to complete the task, often influence crowdworkers to follow specific patterns, which are then …

Content truncated to 500 words