Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks

Proceedings of the 28th International Conference on Computational Linguistics, pages 3436–3440 Barcelona, Spain (Online), December 8-13, 2020 3436 Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks Lichao Sun1∗, Congying Xia2∗, Wenpeng Yin3, Tingting Liang4, Philip S. Yu2, Lifang He1 1 Lehigh University; 2 University of Illinois at Chicago; 3 Salesforce Research; 4 Hangzhou Dianzi University james.lichao.sun@gmail.com; {cxia8, psyu}@uic.edu; wyin@salesforce.com; liangtt@hdu.edu.cn; lih319@lehigh.edu Abstract Mixup (Zhang et al., 2017) is a latest data augmentation technique that linearly interpolates input examples and the corresponding labels. It has shown strong effectiveness in image classification by interpolating images at the pixel level. Inspired by this line of research, in this paper, we explore: i) how to apply mixup to natural language processing tasks since text data can hardly be mixed in the raw format; ii) if mixup is still effective in transformer-based learning models, e.g., BERT. To achieve the goal, we incorporate mixup to transformer-based pre-trained architecture, named “mixup-transformer”, for a wide range of NLP tasks while keeping the whole end-to-end training system. We evaluate the proposed framework by running extensive experiments on the GLUE benchmark. Furthermore, we also examine the performance of mixup-transformer in low-resource scenarios by reducing the training data with a certain ratio. Our studies show that mixup is a domain-independent data augmentation technique to pre-trained language models, resulting in significant performance improvement for transformer-based models. 1 Introduction Deep learning has shown outstanding performance in the field of natural language processing (NLP). Recently, transformer-based methods (Devlin et al., 2018; Yang et al., 2019) have achieved state-of- the-art performance across a wide variety of NLP tasks1. However, these models highly rely on the availability of large amounts of annotated data, which is expensive and labor-intensive. To solve the data scarcity problem, data augmentation is commonly used in NLP tasks. For example, Wei and Zou (2019) investigated language transformations like insertion, deletion and swap. Several works (Malandrakis et al., 2019; Yoo et al., 2019; Xia et al., 2020b; Xia et al., 2020a) utilized variational autoencoders (VAEs) (Kingma and Welling, 2013) to generate more raw inputs. Nevertheless, these methods often rely on some extra knowledge to guarantee the quality of new inputs, and they have to be working in a pipeline. Zhang et al. (2017) proposed mixup, a domain-independent data augmentation technique that linearly interpolates image inputs on the pixel-based feature space. Guo et al. (2019) tried mixup in CNN (LeCun et al., 1998) and LSTM (Hochreiter and Schmidhuber, 1997) for text applications. Despite effectiveness, they conducted mixup only on the fixed word embedding level like Zhang et al. (2017) did in image classification. Two questions arise, therefore: (i) how to apply mixup to NLP tasks if text data cannot be mixed in the raw format? Apart from the embedding feature space, what other representation spaces can be constituted and used? ii) whether or not mixup can boost the state-of-the-art further in transformer-based learning models, such as BERT (Devlin et al., 2018). To answer these questions, we stack a mixup layer over the final hidden layer …

Content truncated to 500 words