A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation

Findings of the Association for Computational Linguistics: ACL 2022, pages 202 - 217 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Analyzing Dynamic Adversarial Training Data in the Limit Eric Wallace1∗ Adina Williams2† Robin Jia2,3† Douwe Kiela2† 1UC Berkeley 2Facebook AI Research 3USC ericwallace@berkeley.edu dkiela@fb.com Abstract To create models that are robust across a wide range of test inputs, training datasets should include diverse examples that span numerous phenomena. Dynamic adversarial data collec- tion (DADC), where annotators craft examples that challenge continually improving models, holds promise as an approach for generating such diverse training sets. Prior work has shown that running DADC over 1–3 rounds can help models fix some error types, but it does not necessarily lead to better generaliza- tion beyond adversarial test data. We argue that running DADC over many rounds maxi- mizes its training-time benefits, as the differ- ent rounds can together cover many of the task-relevant phenomena. We present the first study of longer-term DADC, where we collect 20 rounds of NLI examples for a small set of premise paragraphs, with both adversarial and non-adversarial approaches. Models trained on DADC examples make 26% fewer errors on our expert-curated test set compared to mod- els trained on non-adversarial data. Our analy- sis shows that DADC yields examples that are more difficult, more lexically and syntactically diverse, and contain fewer annotation artifacts compared to non-adversarial examples. 1 Introduction Traditional crowdsourcing methods often yield datasets that lack diversity, contain spurious cor- relations, and are easy for existing models (Guru- rangan et al., 2018; Poliak et al., 2018; Geva et al., 2019; Ko et al., 2020; Potts et al., 2021). Training on such examples can lead to models that reach deceptively high accuracy on in-distribution test data, yet fail on challenge sets (Naik et al., 2018; Glockner et al., 2018; Gardner et al., 2020), input perturbations (Wallace et al., 2019; Kaushik et al., 2020), and distribution shifts (Talmor and Berant, 2019; Hendrycks et al., 2020). ∗Work done while an intern at Facebook AI Research. †Equal contribution 0 5 10 15 20 After Training on N Rounds 70 80 90 Test Accuracy (%) Human Accuracy Data Collection Method Non-Adversarial Static Adversarial Dynamic Adversarial Figure 1: Model accuracy on our expert-curated test set when training on data collected from three differ- ent methods. Standard non-adversarial data collection is more effective than adversarial data collection in the short-term. However, in the long term, adversarial data collection statistically significantly outperforms stan- dard data, especially when the data is collected using a dynamic model that is updated after each round. Dynamic adversarial data collection (DADC) holds promise as an approach to mitigate these training set problems. In DADC, humans are tasked with creating examples that fool state-of-the-art models but are answerable by humans. Crucially, DADC is dynamic in that data collection is repeated over many rounds with a stream of ever-improving models-in-the-loop. As models improve, annota- tors are incentivized to craft new types of examples that challenge the latest models. In the limit, this …

Content truncated to 500 words