Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9029–9043 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9029 Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification Shuhuai Ren1, Jinchao Zhang3, Lei Li1, Xu Sun1,2, Jie Zhou3 1MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2Center for Data Science, Peking University 3Pattern Recognition Center, WeChat AI, Tencent Inc, China {shuhuai_ren, lilei}@stu.pku.edu.cn, xusun@pku.edu.cn {dayerzhang, jiezhou}@tencent.com Abstract Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations. Traditional methods first devise task-specific operations such as Synonym Substitute, then preset the corresponding parameters such as the substitution rate artificially, which require a lot of prior knowledge and are prone to fall into the sub-optimum. Besides, the number of editing operations is limited in the previ- ous methods, which decreases the diversity of the augmented data and thus restricts the per- formance gain. To overcome the above limi- tations, we propose a framework named Text AutoAugment (TAA) to establish a composi- tional and learnable paradigm for data aug- mentation. We regard a combination of vari- ous operations as an augmentation policy and utilize an efficient Bayesian Optimization al- gorithm to automatically search for the best policy, which substantially improves the gen- eralization capability of models. Experiments on six benchmark datasets show that TAA boosts classification accuracy in low-resource and class-imbalanced regimes by an average of 8.8% and 9.7%, respectively, outperforming strong baselines.1 1 Introduction Model performance on Natural Language Process- ing (NLP) tasks, such as text classification, of- ten heavily depends on the size and the quality of the training data. However, it is time-consuming and labor-intensive to obtain sufficient training in- stances, and models face low-resource regimes most of the time. Data augmentation (Simard et al., 1996; Szegedy et al., 2015; Wei and Zou, 2019) aims to enlarge the training dataset by synthesiz- ing additional distinct and label-invariant instances based on raw instances to improve the model per- 1Our code is available at https://github.com/ lancopku/text-autoaugment 0.0 0.2 0.4 0.6 0.8 Magnitude of Operation 50 55 60 65 70 75 80 Test Accuracy (%) Random Swap Random Delete TF-IDF Insert TF-IDF Substitute WordNet Substitute Figure 1: Test accuracy on IMDB dataset with different editing operations and parameters for data augmenta- tion. The operations and parameters without elaborate selection lead to a degrade of performance. formance. Data augmentation approaches demon- strate their superiority in many scenarios, espe- cially when data resources are insufficient (Hu et al., 2019b; Anaby-Tavor et al., 2020; Chen et al., 2021) and class-imbalanced (Ren et al., 2018; Hu et al., 2019b). Previous data augmentation methods can be roughly divided into two categories: generation- based (Sennrich et al., 2016; Imamura et al., 2018; Kobayashi, 2018; Anaby-Tavor et al., 2020; Quteineh et al., 2020) and editing-based meth- ods (Wei and Zou, 2019; Xie et al., 2020). Generation-based methods utilize conditional gen- eration models (Sutskever et al., 2014; Ott et al., 2019) to synthesize the paraphrases (Kumar et al., …

Content truncated to 500 words