BatchMixup: Improving Training by Interpolating Hidden States of the Entire Mini-batch

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4908–4912 August 1–6, 2021. ©2021 Association for Computational Linguistics 4908 BATCHMIXUP: Improving Training by Interpolating Hidden States of the Entire Mini-batch Wenpeng Yin, Huan Wang, Jin Qu and Caiming Xiong Salesforce Research wyin@salesforce.com Abstract Usually, we train a neural system on a se- quence of mini-batches of labeled instances. Each mini-batch is composed of k samples, and each sample will learn a representation vector. MIXUP implicitly generates synthetic samples through linearly interpolating inputs and their corresponding labels of random sam- ple pairs in the same mini-batch. This means that MIXUP only generates new points on the edges connecting every two original points in the representation space. We observed that the new points by the standard MIXUP cover pretty limited regions in the entire space of the mini-batch. In this work, we propose BATCHMIXUP—improving the model learn- ing by interpolating hidden states of the en- tire mini-batch. BATCHMIXUP can gener- ate new points scattered throughout the space corresponding to the mini-batch. In experi- ments, BATCHMIXUP shows superior perfor- mance than competitive baselines in improv- ing the performance of NLP tasks while using different ratios of training data. 1 Introduction The study of data augmentation techniques has a long history in the NLP community. Typi- cal data augmentations include synonym replace- ment (Kobayashi, 2018), back-translation (Fadaee et al., 2017), adding data noise (Xie et al., 2017), etc. Mostly, these techniques are combined with the augmentation-free models in pipeline. MIXUP (Zhang et al., 2018) is able to augment the data by linearly combining each two examples by their hidden representations, keeping the whole system trained in end-to-end. MIXUP has shown effectiveness in a range of NLP tasks (Sun et al., 2020). Nevertheless, it has two drawbacks. First, MIXUP generates new points merely and exactly on the connecting edges of random point pairs; these new points cover pretty limited region in the representation space of the mini-batch. Second, the training of a system equipped with MIXUP is considerably inefficient— generally, MIXUP slows down the training by n times if it generates n new points for each original point pair. In this work, we propose BATCHMIXUP, an improved mixup paradigm that generates new points scattered uniformly throughout the whole representation region of the mini-batch. Specifi- cally, within a mini-batch, each example and its label will first learn a representation vector respec- tively, BATCHMIXUP then generates n new points (including a new input representations and a new la- bel representation) simultaneously by non-linearly interpolating all the examples in the same mini- batch. The new n points are expected to better identify the space represented by the mini-batch. Finally, the n mixed points will act as one batch to update the model. Our model BATCHMIXUP, as a batch-wise non- linear MIXUP, shows advantages in two aspects. (i) Compared with the standard MIXUP, BATCH- MIXUP further improves the representation learn- ing in solving downstream NLP tasks, yielding bet- ter performance. (ii) BATCHMIXUP works much more efficient than the …

Content truncated to 500 words