mixSeq: A Simple Data Augmentation Methodfor Neural Machine Translation

Proceedings of the 18th International Conference on Spoken Language Translation, pages 192–197 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 192 mixSeq: A Simple Data Augmentation Method for Neural Machine Translation Xueqing Wu 1, Yingce Xia 2, Jinhua Zhu 1, Lijun Wu 2, Shufang Xie 2, Tao Qin 2 1 University of Science and Technology of China 2Microsoft Research Asia jwuwuwu24@gmail.com, teslazhu@mail.ustc.edu.cn {yingce.xia,lijuwu,shufxi,taoqin}@microsoft.com Abstract Data augmentation, which refers to manipu- lating the inputs (e.g., adding random noise, masking specific parts) to enlarge the dataset, has been widely adopted in machine learn- ing. Most data augmentation techniques op- erate on a single input, which limits the di- versity of the training corpus. In this paper, we propose a simple yet effective data aug- mentation technique for neural machine trans- lation, mixSeq, which operates on multiple in- puts and their corresponding targets. Specifi- cally, we randomly select two input sequences, concatenate them together as a longer input as well as their corresponding target sequences as an enlarged target, and train models on the augmented dataset. Experiments on nine ma- chine translation tasks demonstrate that such a simple method boosts the baselines by a non- trivial margin. Our method can be further com- bined with single-input based data augmenta- tion methods to obtain further improvements. 1 Introduction Data augmentation, which enlarges the training cor- pus by manipulating the inputs through given rules, has been widely used in machine learning tasks. For image classification, there are various data augmentation methods, including cropping, flip- ping, rotating,cut-out (DeVries and Taylor, 2017), etc. For natural language processing (briefly, NLP), similar data augmentation methods also exist, like randomly swapping words (Lample et al., 2018a), dropping words (Iyyer et al., 2015), and masking specific words (Xie et al., 2017). With data aug- mentation, the main content of the input is not af- fected but the noise is introduced so as to increase the diversity of the training set. The effectiveness of above data augmentation methods has been ver- ified by their strong performance improvements in both image processing and NLP tasks. For ex- ample, with the combination of data augmentation and meta-learning, state-of-the-art result of image classification is achieved (Cubuk et al., 2019). Most existing data augmentation methods take one sample from the training set as input, which might limit the scope and diversity of the train- ing corpus. Mixup (Zhang et al., 2018) is a re- cently proposed data augmentation method, where two samples from the training corpus are lever- aged to build a synthetic sample. Specifically, let x1, x2 denote two images from the training set, and y1, y2 denote their corresponding labels. The syn- thetic data (λx1 + (1 − λ)x2, λy1 + (1 − λ)y2) is introduced to the augmented dataset, where λ is randomly generated. Such a strategy is further enhanced in follow-up works (Zhang et al., 2019; Berthelot et al., 2019). Pair sampling (Inoue, 2018) is another data augmentation method where the synthetic sample is built as (0.5x1 + 0.5x2, …

Content truncated to 500 words