On Transferability of Prompt Tuning for Natural Language Processing

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3949 - 3969 July 10-15, 2022 ©2022 Association for Computational Linguistics On Transferability of Prompt Tuning for Natural Language Processing Yusheng Su1,3∗, Xiaozhi Wang1,3∗, Yujia Qin1,3, Chi-Min Chan1,3, Yankai Lin4, Huadong Wang1,3, Kaiyue Wen1,3, Zhiyuan Liu1,3† , Peng Li2‡, Juanzi Li1,3, Lei Hou1,3, Maosong Sun1,3+, Jie Zhou4 1Department of Computer Science and Technology, BNRist; 2Institute for AI Industry Research (AIR); 3Institute for Artificial Intelligence, Tsinghua University, Beijing, 100084, China 4Pattern Recognition Center, WeChat AI, Tencent Inc, China. {suys19,wangxz20}@mails.tsinghua.edu.cn Abstract Prompt tuning (PT) is a promising parameter- efficient method to utilize extremely large pre-trained language models (PLMs), which can achieve comparable performance to full- parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intu- itively, knowledge transfer can help to improve the efficiency. To explore whether we can im- prove PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks on the same PLM and also to other PLMs with a cross-model projector trained on similar tasks; (2) when used as initialization, trained soft prompts of similar tasks and projected prompts of other PLMs can significantly accelerate train- ing and also improve the performance of PT. Moreover, to explore what decides prompt transferability, we investigate various trans- ferability indicators and find that the over- lapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts stimulate PLMs is essen- tial. Our findings show that prompt trans- fer is promising for improving PT, and fur- ther research shall focus more on prompts’ stimulation to PLMs. The source code can be obtained from https://github.com/ thunlp/Prompt-Transferability. 1 Introduction Pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018) have achieved great performance on vari- ous natural language processing (NLP) tasks (Han et al., 2021). Recently, people have found that ∗ The first two authors contributed equally. † Corresponding author: Z.Liu and M.Sun. ‡ Partly done while P.Li was working at Tencent. Initialization Cross-Task Transfer Source Task Target Task Cross-Model Transfer Prompt I am glad to… Task PLM Source PLM Reuse / Initialize This place is… Prompt Reuse / Initialize Target PLM Prompt I am glad to… I am glad to… Prompt Projector Figure 1: We explore prompt transferring across dif- ferent tasks (cross-task) and PLMs (cross-model) with directly reusing prompts and initializing prompt tuning. extremely large PLMs can achieve remarkable im- provements, and various large PLMs are continu- ally developed (Brown et al., 2020; Raffel et al., 2020; Zhang et al., 2021; Zeng et al., 2021; Wei et al., 2021; Sun et al., 2021), which contain up to hundreds of billions of parameters. Considering the extremely large scale of these state-of-the-art PLMs, conventional full-parameter fine-tuning methods become extremely …

Content truncated to 500 words