Good-Enough Compositional Data Augmentation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556–7566 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7556 Good-Enough Compositional Data Augmentation Jacob Andreas MIT CSAIL jda@mit.edu Abstract We propose a simple data augmentation pro- tocol aimed at providing a compositional in- ductive bias in conditional and unconditional sequence models. Under this protocol, syn- thetic training examples are constructed by tak- ing real training examples and replacing (pos- sibly discontinuous) fragments with other frag- ments that appear in at least one similar envi- ronment. The protocol is model-agnostic and useful for a variety of tasks. Applied to neural sequence-to-sequence models, it reduces error rate by as much as 87% on diagnostic tasks from the SCAN dataset and 16% on a seman- tic parsing task. Applied to n-gram language models, it reduces perplexity by roughly 1% on small corpora in several languages. 1 Introduction This paper proposes a rule-based data augmenta- tion protocol for sequence modeling. Our approach aims to supply a simple and model-agnostic bias toward compositional reuse of previously observed sequence fragments in novel environments. Con- sider a language modeling task in which we wish to estimate a probability distribution over a family of sentences with the following finite sample as training data: (1) a. The cat sang. b. The wug sang. c. The cat daxed. In language processing problems, we often want models to generalize beyond this dataset and infer that (2a) is also probable but (2b) is not: (2) a. The wug daxed. b. * The sang daxed. This generalization amounts to an inference about syntactic categories (Clark, 2000). Because cat and wug are interchangeable in (1a) and (1b), they are also likely interchangeable elsewhere; cat and sang are not similarly interchangeable. Human learners make judgments like (2) about novel lexical items (Berko, 1958) and fragments of novel languages (Lake et al., 2019). But we do not expect such judgments from unstructured generative models trained to maximize the likelihood of the training data in (1). A large body of work in natural language pro- cessing provides generalization to data like (2a) by adding structure to the learned predictor (Chelba and Jelinek, 1998; Chiang, 2005; Dyer et al., 2016). On real-world datasets, however, such models are typically worse than “black-box” function approx- imators like neural networks, even for black-box models that fail to place probability mass on ei- ther example in (2) given small training sets like (1) (Melis et al., 2018). To the extent that we be- lieve (2a) to capture an important inductive bias, we would like to find a way of softly encouraging it without tampering with the structure of predictors that work well at scale. In this paper, we introduce a procedure for generating synthetic training ex- amples by recombining real ones, such that (2a) is assigned non-negligible probability because it already appears in the training dataset. The basic operation underlying our proposal (which we call GECA, for “good-enough compo- sitional augmentation”) is depicted in Figure 1: …

Content truncated to 500 words