PromptDA: Label-guided Data Augmentation for Prompt-based Few Shot Learners

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 562–574 May 2-6, 2023 ©2023 Association for Computational Linguistics PROMPTDA : Label-guided Data Augmentation for Prompt-based Few Shot Learners Canyu Chen Illinois Institute of Technology cchen151@hawk.iit.edu Kai Shu Illinois Institute of Technology kshu@iit.edu Abstract Recent advances in large pre-trained language models (PLMs) lead to impressive gains on nat- ural language understanding (NLU) tasks with task-specific fine-tuning. However, directly fine-tuning PLMs heavily relies on sufficient labeled training instances, which are usually hard to obtain. Prompt-based tuning on PLMs has shown to be powerful for various down- stream few-shot tasks. Existing works studying prompt-based tuning for few-shot NLU tasks mainly focus on deriving proper label words with a verbalizer or generating prompt tem- plates to elicit semantics from PLMs. In addi- tion, conventional data augmentation strategies such as synonym substitution are also widely adopted in low-resource scenarios. However, the improvements they bring to prompt-based few-shot learning have been demonstrated to be marginal. Thus, an important research question arises as follows: how to design effective data augmentation methods for prompt-based few- shot tuning? To this end, considering the label semantics are essential in prompt-based tuning, we propose a novel label-guided data augmen- tation framework PROMPTDA, which exploits the enriched label semantic information for data augmentation. Extensive experiment re- sults on few-shot text classification tasks show that our proposed framework achieves superior performances by effectively leveraging label semantics and data augmentation for natural language understanding. 1 Introduction Pre-trained language models (PLMs) have shown promising performances in various applications such as text classification (Yang et al., 2019), docu- ment summarization (Zhang et al., 2020a), question answering (Mirzaee et al., 2021). The recent ad- vancement of prompt-based tuning has shown a significant improvement over normal fine-tuning on various few-shot tasks (Brown et al., 2020). Typically, the prompt-based tuning paradigm trans- [CLS] neat movie to watch. [SEP] [CLS] nice film to watch. [SEP] [CLS] nice movie to watch. [SEP] (a) Conventional augmentation method (Synonym Substitution) [CLS] nice movie to watch. [SEP] (b) Our augmentation approach for Prompt Tuning good great fine PLM [CLS] nice movie to watch. [SEP] [CLS] nice movie to watch. [SEP] PLM [CLS] nice movie to watch. [SEP] [CLS] nice movie to watch. [SEP] Figure 1: The basic comparison of conventional data augmentation methods and our proposed augmentation framework PROMPTDA. {good, great, fine} are the label words for prompt tuning. Conventional DA constructs instances for augmentation. But PROMPTDA conducts instance-label pairs for augmentation. forms a NLU task into a masked language model- ing (MLM) problem. For example, in sentiment analysis, an original sentence “nice movie to watch." can be augmented with a template “It is [MASK]" as the input x. Each class (e.g., POSITIVE) is represented by a label word (e.g., good) selected by a verbalizer from the vocabu- lary (Schick and Schütze, 2021). The prediction of the class POSITIVE is based on the probability of the [MASK] being filled with the token good. In addition, conventional data …

Content truncated to 500 words