PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4242 - 4255 May 22-27, 2022 c⃝2022 Association for Computational Linguistics PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks Yufei Wang1∗, Can Xu2, Qingfeng Sun2, Huang Hu2, Chongyang Tao2 Xiubo Geng2 and Daxin Jiang2† Macquarie University, Sydney, Australia1 Microsoft Corporation, Beijing, China2 yufei.wang@students.mq.edu.au {caxu,qins,huahu,chongyang.tao,xigeng,djiang}@microsoft.com Abstract This paper focuses on the Data Augmenta- tion for low-resource Natural Language Under- standing (NLU) tasks. We propose Prompt- based Data Augmentation model (PromDA) which only trains small-scale Soft Prompt (i.e., a set of trainable vectors) in the frozen Pre-trained Language Models (PLMs). This avoids human effort in collecting unlabeled in- domain data and maintains the quality of gen- erated synthetic data. In addition, PromDA generates synthetic data via two different views and filters out the low-quality data us- ing NLU models. Experiments on four bench- marks show that synthetic data produced by PromDA successfully boost up the perfor- mance of NLU models which consistently out- perform several competitive baseline models, including a state-of-the-art semi-supervised model using unlabeled in-domain data. The synthetic data from PromDA are also comple- mentary with unlabeled in-domain data. The NLU models can be further improved when they are combined for training. 1 Introduction Deep neural networks often require large-scale high-quality labeled training data to achieve state- of-the-art performance (Bowman et al., 2015). However, constructing labeled data could be chal- lenging in many scenarios (Feng et al., 2021). In this paper, we study the low-resource Natural Lan- guage Understanding (NLU) tasks, including sen- tence classification and sequence labelling tasks, where only small labeled data is available. Previous works often produce extra “labeled data” for the NLU models to learn. Wang et al. (2021a) deploys the self-training framework to produce pseudo la- belled training data from unlabeled in-domain data which could be expensive to obtain. Xu et al. (2021) ∗Work done during the internship at Microsoft STCA. †Corresponding author has shown that extracting domain-specific unla- beled data from the general corpus is not trivial. Wei and Zou (2019); Dai and Adel (2020) expand the original small training data using automatic heuristic rules, such as randomly synonyms re- placement, which effectively creates new training instances. However, these processes may distort the text, making the generated syntactic data gram- matically and semantically incorrect. To solve the above dilemma, many existing works (Ding et al., 2020; Yang et al., 2020; Anaby- Tavor et al., 2020) resort to applying Language Models (LMs) or Pre-trained Language Models (PLMs) for data augmentation in a low-resource setting. Given the labeled data, one can directly fine-tune PLMs to generate new synthetic data with- out additional human effort. However, we argue that, in the low-resource NLU tasks, directly fine- tuning all parameters of PLMs with small train- ing data (especially when there are less than 100 samples) could result in over-fitting and PLMs sim- ply memorizes the training instances. As a result, the generated synthetic data could be very simi- lar to the original …

Content truncated to 500 words