DISCO: Distilling Counterfactuals with Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5514–5528 July 9-14, 2023 ©2023 Association for Computational Linguistics DISCO: Distilling Counterfactuals with Large Language Models Zeming Chen†∗ Qiyue Gao‡∗ Antoine Bosselut† Ashish Sabharwal‡ Kyle Richardson‡ † Natural Language Processing Lab, EPFL, Lausanne, Switzerland {zeming.chen, antoine.bosselut}@epfl.ch ‡ Allen Institute for AI, Seattle, U.S.A. {bertg, kyler, ashishs}@allenai.org Abstract Models trained with counterfactually aug- mented data learn representations of the causal structure of tasks, enabling robust generaliza- tion. However, high-quality counterfactual data is scarce for most tasks and not easily generated at scale. When crowdsourced, such data is typi- cally limited in scale and diversity; when gen- erated using supervised methods, it is compu- tationally expensive to extend to new counter- factual dimensions. In this work, we introduce DISCO (DIStilled COunterfactual Data), a new method for automatically generating high- quality counterfactual data at scale. DISCO engineers prompts to generate phrasal pertur- bations with a large general language model. Then, a task-specific teacher model filters these generations to distill high-quality counterfac- tual data. While task-agnostic, we apply our pipeline to the task of natural language infer- ence (NLI) and find that on challenging evalua- tions such as the NLI stress test, comparatively smaller student models trained with DISCO- generated counterfactuals are more robust (6% absolute) and generalize better across distribu- tions (2%) compared to models trained with- out data augmentation. Furthermore, DISCO- augmented models are 10% more consistent be- tween counterfactual pairs on three evaluation sets, demonstrating that DISCO augmentation enables models to more reliably learn causal representations. Our repository are available at: https://github.com/eric11eca/disco 1 Introduction Despite the tremendous progress made in NLP on a wide range of reasoning tasks (Wang et al., 2018, 2019a; Xu et al., 2020), dataset biases continue to be a formidable challenge for robust model de- velopment (Gururangan et al., 2018; Poliak et al., 2018; Kaushik and Lipton, 2018; Tsuchiya, 2018; Liu et al., 2020b; Du et al., 2022). Counterfactual ∗ Work done while at the Allen Institute for AI. Equal contribution Span Extraction Counterfactual Data Easy Ambigious Hard Counterfactual Over-generation Data Selection Seed Dataset Original: A young girl looks up as she rides on a merry-go-round. Spans: A young girl looks up as she rides on a merry-go-round. Counterfactual: A young girl looks up at a very tall roller coaster with an eager and excited look on her face. General LLM (GPT-3) Specialized Teacher Model Filtering Student Model Data Augmentation Figure 1: Overview of our counterfactual data distilla- tion process (DISCO) using a large language model. data augmentation (CAD) (Kaushik et al., 2019) is one general approach to improve model robustness by training on edited instances that systematically alter the critical or causally salient parts of dataset instances that contributes to the label assignment. To date, two main approaches have been pursued as part of these efforts: human-centered approaches, where edits are obtained through direct human an- notation and crowdsourcing (Kaushik et al., 2019; Khashabi et al., 2020; Gardner et …

Content truncated to 500 words