A Multi-Level Optimization Framework for End-to-End Text Augmentation

A Multi-Level Optimization Framework for End-to-End Text Augmentation Sai Ashish Somayajula UC San Diego, USA ssomayaj@ucsd.edu Linfeng Song Tencent AI Lab, USA lfsong@tencent.com Pengtao Xie∗ UC San Diego, USA p1xie@eng.ucsd.edu Abstract Text augmentation is an effective technique in alleviating overfitting in NLP tasks. In ex- isting methods, text augmentation and down- stream tasks are mostly performed separately. As a result, the augmented texts may not be optimal to train the downstream model. To address this problem, we propose a three-level optimization framework to perform text aug- mentation and the downstream task end-to- end. The augmentation model is trained in a way tailored to the downstream task. Our framework consists of three learning stages. A text summarization model is trained to per- form data augmentation at the first stage. Each summarization example is associated with a weight to account for its domain difference with the text classification data. At the second stage, we use the model trained at the first stage to perform text augmentation and train a text classification model on the augmented texts. At the third stage, we evaluate the text classification model trained at the second stage and update weights of summarization exam- ples by minimizing the validation loss. These three stages are performed end-to-end. We evaluate our method on several text classifica- tion datasets where the results demonstrate the effectiveness of our method. Code is available at https://github.com/Sai-Ashish /End-to-End-Text-Augmentation. 1 Introduction Data augmentation (Sennrich et al., 2015; Fadaee et al., 2017; Wei and Zou, 2019) is an effec- tive technique for mitigating the deficiency of training data and preventing overfitting. In natu- ral language processing, many data augmentation methods have been proposed, such as back transla- tion (Sennrich et al., 2015), synonym replacement (Wang and Yang, 2015), random insertion (Wei and Zou, 2019), and so on. In existing approaches, ∗Corresponding author data augmentation and downstream tasks are per- formed separately: Augmented texts are created first, then they are used to train a downstream model. The downstream task does not influence the augmentation process. As a result, the aug- mented texts may not be optimal for training the downstream model. In this paper, we aim to address this problem. We propose an end-to-end learning framework based on multi-level optimization (Feurer et al., 2015), which performs data augmentation and downstream tasks in a unified manner where not only augmented texts influence the training of the downstream model, but also the performance of downstream task affects how data augmentation is performed. In our framework, we use a text summarization (Gambhir and Gupta, 2017) model to perform data augmentation. Given an original text t with class label c, we feed t into the summarization model to generate a summary s. We set the class label of s to be c. (s, c) is treated as an augmented text-label pair of (t, c). The motivation of using a summarization model for text augmentation is two-fold. First, the major semantics of an original text is preserved in its summary; therefore, it is sensible …

Content truncated to 500 words