Generative Data Augmentation for Commonsense Reasoning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1008–1025 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1008 Generative Data Augmentation for Commonsense Reasoning Yiben Yang♠ Chaitanya Malaviya†Π Jared Fernandez♠♥ Swabha Swayamdipta† Ronan Le Bras† Ji-Ping Wang♠ Chandra Bhagavatula† Yejin Choi†♦ Doug Downey♠† ♠Northwestern University, Evanston, IL, USA †Allen Institute for Artificial Intelligence, Seattle, WA, USA Π University of Pennsylvania, Philadelphia, PA, USA ♥ Carnegie Mellon University, Pittsburgh, PA, USA ♦University of Washington, Seattle, WA, USA {yiben.yang@,jared.fern@u.,jzwang@}northwestern.edu {chaitanyam,swabhas,ronanlb,chandrab,yejinc,dougd}@allenai.org Abstract Recent advances in commonsense reasoning depend on large-scale human-annotated train- ing sets to achieve peak performance. How- ever, manual curation of training sets is ex- pensive and has been shown to introduce an- notation artifacts that neural models can read- ily exploit and overfit to. We propose a novel generative data augmentation technique, G-DAUGc, that aims to achieve more accu- rate and robust learning in a low-resource set- ting. Our approach generates synthetic exam- ples using pretrained language models, and selects the most informative and diverse set of examples for data augmentation. On ex- periments with multiple commonsense reason- ing benchmarks, G-DAUGc consistently out- performs existing data augmentation methods based on back-translation, establishing a new state-of-the-art on WINOGRANDE, CODAH, and COMMONSENSEQA, and also enhances out-of-distribution generalization, proving to be more robust against adversaries or per- turbations. Our analysis demonstrates that G-DAUGc produces a diverse set of fluent training examples, and that its selection and training approaches are important for perfor- mance. 1 Introduction While recent advances in large-scale neural lan- guage models (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Raffel et al., 2019) have led to strong performance on several commonsense rea- soning benchmarks (Talmor et al., 2019; Lv et al., 2020; Sakaguchi et al., 2020), their accuracy by and large depends on the availability of large-scale human-authored training data. However, crowd- sourcing examples at scale for each new task and domain can be prohibitively expensive. Moreover, human-authored data has been shown to exhibit an- notation artifacts (Gururangan et al., 2018; Agrawal Figure 1: Example of a selected high-quality generated example compared to a human-authored example from the WINOGRANDE dataset. Composing commonsense questions can require creativity. et al., 2018; Schwartz et al., 2017), leading to mod- els with considerably weaker performance on out- of-distribution samples (Jia and Liang, 2017; Be- linkov and Bisk, 2017; Iyyer et al., 2018). A candidate solution that has shown promise in other tasks, such as reading comprehension, is to augment a human-authored training set with a large set of synthetically-generated examples (Zhou et al., 2017; Du et al., 2017; Zhao et al., 2018a). But, generating synthetic examples for commonsense reasoning poses a unique challenge. In reading comprehension, for instance, the goal of data augmentation is to generate questions that are directly answerable by a given reference passage. In contrast, answering commonsense questions re- lies on commonsense notions that are seldom stated explicitly (Gordon and Van Durme, 2013; Forbes and Choi, 2017), and authoring such questions can require …

Content truncated to 500 words