Data Augmentation for Text Generation Without Any Augmented Data

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2223–2237 August 1–6, 2021. ©2021 Association for Computational Linguistics 2223 Data Augmentation for Text Generation Without Any Augmented Data Wei Bi∗ Huayang Li∗ Jiacheng Huang Tencent AI Lab, Shenzhen, China {victoriabi,alanili,eziohuang}@tencent.com Abstract Data augmentation is an effective way to im- prove the performance of many neural text gen- eration models. However, current data aug- mentation methods need to define or choose proper data mapping functions that map the original samples into the augmented samples. In this work, we derive an objective to for- mulate the problem of data augmentation on text generation tasks without any use of aug- mented data constructed by specific mapping functions. Our proposed objective can be effi- ciently optimized and applied to popular loss functions on text generation tasks with a con- vergence rate guarantee. Experiments on five datasets of two text generation tasks show that our approach can approximate or even surpass popular data augmentation methods. 1 Introduction End-to-end neural models are generally trained in a data-driven paradigm. Many researchers have pro- posed powerful network structures to fit training data well. It has also become ubiquitous to in- crease the training data amount to improve model performance. Data augmentation is an effective technique to create additional samples in both vi- sion and text classification tasks (Perez and Wang, 2017; Shorten and Khoshgoftaar, 2019; Wei and Zou, 2019), which perturb samples without chang- ing their labels. For text generation tasks, there can be more types of data perturbation to construct augmented samples, including corrupting the in- put text (Xie et al., 2017), the output text (Norouzi et al., 2016; Kurata et al., 2016), or both (Zhang et al., 2020). As such, classification tasks can be re- garded as special cases of generation tasks in terms of incorporating data augmentation techniques, and this work mainly discusses text generation tasks. ∗Equal contribution. The focus of previous work on text data augmen- tation has been to design proper augmentation tech- niques to create augmented samples. Some aug- mentation methods have been proposed for general text tasks. For example, different general replace- ment operations have been explored to edit words in a text sample, ranging from simple look-up ta- bles (Zhang et al., 2015) to pretrained masked lan- guage models (Kobayashi, 2018; Wu et al., 2019). Sennrich et al. (2016) propose to augment text sequences by back-translation. For some gener- ation tasks such as dialogue generation, general augmentation methods may not yield stable im- provements and it requires to carefully incorporate the task property to design useful augmented sam- ples (Zhang et al., 2020). All these methods need to explicitly construct augmented samples, and the data mapping functions from the original samples to the augmented samples are mostly defined apri- ori. This motivates us to raise a question, whether we can skip the step to define or choose proper augmented data mapping functions to accomplish effective data …

Content truncated to 500 words