Rethinking Data Augmentation in Text-to-text Paradigm

Proceedings of the 29th International Conference on Computational Linguistics, pages 1157–1162 October 12–17, 2022. 1157 Rethinking Data Augmentation in Text-to-text Paradigm Yanan Chen Dept. of Physics and Computer Science Wilfrid Laurier University chen0040@mylaurier.ca Yang Liu Dept. of Physics and Computer Science Wilfrid Laurier University yangliu@wlu.ca Abstract As manually labelling data can be costly, some recent studies tend to augment the training data for improving the generalization power of ma- chine learning models, known as data aug- mentation (DA). With the arise of pre-trained language models (PLMs), some recent works on DA try to synthesize new samples benefit- ing from the knowledge learned from PLM’s pre-training. Along the same direction, we in this paper propose to integrate text-to-text lan- guage models and construct a new two-phase framework for augmentation: 1) a fine-tuning phase where PLMs are well adapted to down- stream classification with the help of two novel schemes, and 2) a generation phase where the fine-tuned models are leveraged to create new samples for performance lifting. This paradigm opens up a new way of designing fine- tuning scheme to better serve DA in an easy-to- implement manner, and can be easily extended to other desired tasks. We evaluate our pro- posal on two public classification datasets and demonstrate its effectiveness with remarkable gains. 1 Introduction Due to the unique challenges of natural language processing tasks, there is no one-size-fits-all DA so- lution. Most early attempts are based on token ma- nipulation (Wei and Zou, 2019; Kobayashi, 2018; Wu et al., 2019) or paraphrasing (Sennrich et al., 2016), and the boost is limited or marginal, or even diminishing or adverse especially given original training corpus is relatively sufficient or the back- bone classifiers are PLM based, such as BERT or RoBERTa (Longpre et al., 2020). Some researchers have shifted attention on ap- plying generative language models(GLMs) for DA (Weng, 2022). Auto-regressive generation LMs such as GPT2 (Radford et al., 2019) are capable of generating text with high fluency and diversity, and therefore could serve as generators to synthe- size new samples required by classification model training. However, most existing GLM-based DA solutions have some drawbacks. First, they fine- tune GLMs on the training corpus of limited ca- pacity (Kumar et al., 2020; Anaby-Tavor et al., 2020; Liu et al., 2020), which can be problem- atic and prone to overfitting (Dodge et al., 2020; Phang et al., 2018; Ruder, 2021). Second, how to introduce external colossal online corpus freely available, such as reviews, comments and news to benefit GLMs to better serve DA has not been stud- ied. Third, effective fine-tuning regime customized for data characteristic and structure has rarely been studied. In addition, recent works employing few- shot in-context generation for DA, such as GPT3, in avoidance fine-tuning and reap sparks of clever- ness for automation, suffers from economic costs, latency in usage and lack of reliability (Sahu et al., 2022; Yoo et al., 2021; Wang et al., 2022). To meet above challenges, we explore the poten- tial of using text-to-text …

Content truncated to 500 words