Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3754 - 3767 July 10-15, 2022 ©2022 Association for Computational Linguistics Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants Max Bartolo†‡ Tristan Thrush‡ Sebastian Riedel†‡ Pontus Stenetorp† Robin Jia§‡ Douwe Kiela‡ †UCL §USC ‡Facebook AI Research m.bartolo@cs.ucl.ac.uk Abstract In Dynamic Adversarial Data Collec- tion (DADC), human annotators are tasked with finding examples that models struggle to predict correctly. Models trained on DADC-collected training data have been shown to be more robust in adversarial and out-of-domain settings, and are considerably harder for humans to fool. However, DADC is more time-consuming than traditional data collection and thus more costly per annotated example. In this work, we examine whether we can maintain the advantages of DADC, without incurring the additional cost. To that end, we introduce Generative Annotation Assistants (GAAs), generator-in-the-loop models that provide real-time suggestions that annotators can either approve, modify, or reject entirely. We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection. We demonstrate that GAAs provide significant efficiency benefits with over a 30% annotation speed-up, while leading to over a 5x improve- ment in model fooling rates. In addition, we find that using GAA-assisted training data leads to higher downstream model perfor- mance on a variety of question answering tasks over adversarial data collection. 1 Introduction Natural language processing has become increas- ingly reliant on large datasets obtained using crowd sourcing. However, crowdsourcing as an uncon- strained annotation approach is known to result in machine-exploitable annotator artefacts (Jia and Liang, 2017; Schwartz et al., 2017; Gururangan et al., 2018; Geva et al., 2019), leading to poor out- of-distribution generalisation (Chen et al., 2016; Weissenborn et al., 2017; Yogatama et al., 2019; McCoy et al., 2019). Dynamic Adversarial Data Collection (DADC) aims to address these issues After a slow start to the 2008–09 season, the Bruins won 17 of their next 20 games, leading many to see them as a revival of the "Big Bad Bruins" from the 1970s and 1980s. During the 2009 All-Star Weekend's Skills Competition, captain Zdeno Chara fired the NHL's then-fastest measured "hardest shot" ever, with a clocked in speed of 105.4 mph (169.7 km/h) velocity. (Chara has since broken his own record three times, two of those on the same night.) The number of injured players in the season… A: 105.4 mph Q: What was the fastest shot ever? Q: What was the fastest shot ever in 2009? A: Zdeno Chara GAA QA Figure 1: Example interaction between an annotator and the models in the loop. The annotator selects an answer from the passage, for which the Generative An- notation Assistant (GAA) prompts a question. The an- notator can then freely modify the question and/or an- swer, or generate another prompt. In the adversarial data collection setting, a model-in-the-loop provides predictions with …

Content truncated to 500 words