Sequence-Level Mixed Sample Data Augmentation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5547–5552, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5547 Sequence-Level Mixed Sample Data Augmentation Demi Guo Harvard University dguo@college.harvard.edu Yoon Kim MIT-IBM Watson AI Lab yoonkim@ibm.com Alexander M. Rush Cornell University arush@cornell.edu Abstract Despite their empirical success, neural net- works still have difficulty capturing compo- sitional aspects of natural language. This work proposes a simple data augmentation ap- proach to encourage compositional behavior in neural models for sequence-to-sequence prob- lems. Our approach, SeqMix, creates new synthetic examples by softly combining in- put/output sequences from the training set. We connect this approach to existing techniques such as SwitchOut (Wang et al., 2018) and word dropout (Sennrich et al., 2016), and show that these techniques are all approximating variants of a single objective. SeqMix con- sistently yields approximately 1.0 BLEU im- provement on five different translation datasets over strong Transformer baselines. On tasks that require strong compositional generaliza- tion such as SCAN and semantic parsing, Se- qMix also offers further improvements. 1 Introduction Natural language is thought to be characterized by systematic compositionality (Fodor and Pylyshyn, 1988). A computational model that is able to ex- ploit such systematic compositionality should un- derstand sentences by appropriately recombining subparts that have not been seen together during training. Consider the following example from An- dreas (2020): (1a) She picks the wug up in Fresno. (1b) He puts the cup down in Tempe. Given the above sentences, a model which has learned compositional structure should be able to generalize and understand sentences such as: (2a) She puts the wug down in Fresno. (2b) She picks the wug up in Tempe. In practice, neural models often overfit to long seg- ments of text and fail to generalize compositionally. This work proposes a simple data augmenta- tion strategy for sequence-to-sequence learning, SeqMix, which creates soft synthetic examples by randomly combining parts of two sentences. This prevents models from memorizing long segments and encourages models to rely on compositions of subparts to predict the output. To motivate our approach, consider some example sentences that can be created by combining (1a) and (1b) : (2c) He picks the wug up in Fresno. (2d) She picks the wug up in Tempe. (2e) He picks the cup up in Fresno. (2f) He puts the cup up in Fresno. Instead of enumerating over all possible combina- tions of two sentences, SeqMix crafts a new ex- ample by softly mixing the two sentences via a convex combination of the original examples. This approach can be seen as a sequence-level variant of a broader family of techniques called mixed sam- ple data augmentation (MSDA), which was origi- nally proposed by Zhang et al. (2018) and has been shown to be particularly effective for classification tasks (DeVries and Taylor, 2017; Yun et al., 2019; Verma et al., 2019). We also show that SeqMix shares similarities with word replacement/dropout strategies in machine translation (Sennrich et al., 2016; Wang et al., 2018; Gao et …

Content truncated to 500 words