Text Augmentation in a Multi-Task View

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2888–2894 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2888 Text Augmentation in a Multi-Task View Jason Wei Chengyu Huang Shiqi Xu Soroush Vosoughi ProtagoLabs, {jason,xu}@protagolabs.com International Monetary Fund, huangchengyu24@gmail.com Dartmouth College, soroush@dartmouth.edu Abstract Traditional data augmentation aims to increase the coverage of the input distribution by gener- ating augmented examples that strongly resem- ble original samples in an online fashion where augmented examples dominate training. In this paper, we propose an alternative perspective—a multi-task view (MTV) of data augmentation—in which the primary task trains on original examples and the auxil- iary task trains on augmented examples. In MTV data augmentation, both original and augmented samples are weighted substantively during training, relaxing the constraint that augmented examples must resemble original data and thereby allowing us to apply stronger levels of augmentation. In empirical experiments using four common data augmentation techniques on three bench- mark text classification datasets, we find that the MTV leads to higher and more robust per- formance improvements than traditional aug- mentation. 1 Introduction Most data augmentation techniques aim to generate augmented examples for training that are similar to original data. In computer vision, operations such as flipping, cropping, and color jittering are both widely used and highly effective—it is self- evident that augmented examples closely resemble original data, and so we generate augmented data in an online fashion during each minibatch such that no original, unmodified examples are seen during training (Krizhevsky et al., 2012; Zagoruyko and Komodakis, 2016; Huang et al., 2017). In language, on the other hand, even slight mod- ifications can cause significant semantic changes, and so it is not always clear whether augmented ex- amples resemble original data. Despite this uncer- tainty, many augmentation techniques in NLP still Traditional Data Augmentation • Intuition: Increase coverage of input distribution by using augmented examples for training. • Guideline: Augmented examples should be similar to original data. • Training: Dominated by augmented examples that are generated stochastically. Multi-Task View (MTV) of Data Augmentation • Intuition: Auxiliary task of classifying augmented ex- amples acts as regularization for the primary task of classifying original examples. • Guideline: It might be a good idea for augmented sam- ples to resemble original data, but they can be anything that boosts performance. • Training: Both original and augmented data receive substantive weighting during training. Table 1: Summary of traditional data augmentation ver- sus MTV data augmentation. generate examples stochastically and ignore origi- nal data (Zhang et al., 2015; Sennrich et al., 2016a; Xie et al., 2017; Li et al., 2017; Kobayashi, 2018; Wang et al., 2018). When it is unclear whether augmented examples resemble original data—as is often the case—is it wise to neglect the original training data? Our paper questions this practice by proposing to include original data during training. Specifically, we make two contributions: 1. We propose a multi-task view of data augmen- tation (MTV data augmentation), which trains on …

Content truncated to 500 words