Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6525 - 6540 December 7-11, 2022 ©2022 Association for Computational Linguistics Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models Chaitanya Malaviya and Sudeep Bhatia and Mark Yatskar University of Pennsylvania {cmalaviy, bhatiasu, myatskar}@upenn.edu Abstract Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while ex- pending less effort. While performing anno- tation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of an- notating multiple-choice reading comprehen- sion datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indi- cate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Impor- tantly, heuristic use among annotators deter- mines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authored by annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don’t generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases. 1 Introduction While crowdsourcing is an effective and widely- used data collection method in NLP, it comes with caveats. Crowdsourced datasets have been found to contain artifacts from the annotation process, and models trained on such data, can be brittle and fail to generalize to distribution shifts (Gururangan et al., 2018; Kaushik and Lipton, 2018; McCoy et al., 2019). In this work, we ask whether sys- tematic patterns in annotator behavior influence the quality of collected data. We hypothesize that usage of cognitive heuris- tics, which are mental shortcuts that humans em- ploy in everyday life, can cascade on to data quality and model robustness. For example, an annotator asked to write a question based on a passage might not read the entire passage or might use just one sentence to frame a question. Annotators may seek shortcuts to economize on the amount of time and effort they put into a task. This behavior in annota- tors, characterized by examples that are acceptable but not high-quality, can be problematic. We analyze the extent to which annotators en- gage in various low-effort strategies, akin to cog- nitive heuristics, by tracking indicative features from their annotation data in the form of annota- tor heuristic traces. First, we crowdsource read- ing comprehension questions where we instruct workers to write hard questions. Inspired by re- search on human cognition (Simon, 1956; Tversky and Kahneman, 1974), we identify several heuris- tics that could be employed by annotators for our task, such as satisficing (Simon, 1956), availability (Tversky and Kahneman, 1973) and representative- ness (Kahneman and Tversky, 1972). …

Content truncated to 500 words