TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2792–2802 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2792 TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification Chengyu Wang1∗, Jianing Wang2∗, Minghui Qiu1, Jun Huang1, Ming Gao2† 1 Alibaba Group 2 School of Data Science and Engineering, East China Normal University {chengyu.wcy,minghui.qmh,huangjun.hj}@alibaba-inc.com, lygwjn@gmail.com,mgao@dase.ecnu.edu.cn Abstract Recent studies have shown that prompts im- prove the performance of large pre-trained lan- guage models for few-shot text classification. Yet, it is unclear how the prompting knowledge can be transferred across similar NLP tasks for the purpose of mutual reinforcement. Based on continuous prompt embeddings, we pro- pose TransPrompt, a transferable prompting framework for few-shot learning across simi- lar tasks. In TransPrompt, we employ a multi- task meta-knowledge acquisition procedure to train a meta-learner that captures cross-task transferable knowledge. Two de-biasing tech- niques are further designed to make it more task-agnostic and unbiased towards any tasks. After that, the meta-learner can be adapted to target tasks with high accuracy. Extensive experiments show that TransPrompt outper- forms single-task and cross-task strong base- lines over multiple NLP tasks and datasets. We further show that the meta-learner can ef- fectively improve the performance on previ- ously unseen tasks. TransPrompt also outper- forms strong fine-tuning baselines when learn- ing with full training sets. 1 Introduction Fine-tuning Pre-trained Language Models (PLMs) has become the standard practice to train models for a majority of NLP tasks (Devlin et al., 2019; Liu et al., 2019b; Qiu et al., 2020). To ensure high accu- racy, it is necessary to obtain a sufficient amount of training data for downstream tasks, which is often the bottleneck in low-resource scenarios. The application of ultra-large PLMs such as GPT-3 (Brown et al., 2020) proves that such PLMs can learn to solve a task with very few training sam- ples. Inspired by these works, Gao et al. (2020) pro- pose a prompt-based approach to fine-tune BERT- style PLMs in a few-shot learning setting, which ∗ C. Wang and J. Wang contributed equally to this work. † Corresponding author. adapts PLMs into producing specific tokens cor- responding to each class, instead of learning the prediction head. The effectiveness of prompts has also been shown in Schick and Schütze (2020); Scao and Rush (2021); Schick and Schütze (2021) and others. However, designing high-performing prompts is challenging and requires a very large validation set. To alleviate this problem, Liu et al. (2021) propose continuous prompt embeddings with fully differentiable parameters, avoiding the cumbersome manual prompt engineering process. Despite the remarkable success, we notice that current prompt-based approaches may have a few limitations. For few-shot learning, the performance of downstream tasks is still constrained by the num- ber of training instances. It would be highly de- sirable if the model can acquire the transferable knowledge from similar NLP tasks before it is adapted to specific tasks with few samples. How- ever, it is unclear how the knowledge in prompt encoders and PLMs with prompting techniques …

Content truncated to 500 words