PPT: Pre-trained Prompt Tuning for Few-shot Learning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8410 - 8423 May 22-27, 2022 c⃝2022 Association for Computational Linguistics PPT: Pre-trained Prompt Tuning for Few-shot Learning Yuxian Gu1,3∗, Xu Han2,3∗, Zhiyuan Liu2,3,4, Minlie Huang1,3,4† 1The CoAI group, Tsinghua University, Beijing, China 2The THUNLP group, Tsinghua University, Beijing, China 3Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China 4 Beijing Academy of Artificial Intelligence, BAAI, Beijing, China {guyx21,hanxu17}@mails.tsinghua.edu.cn {liuzy,aihuang}@tsinghua.edu.cn Abstract Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an effi- cient and effective solution for adapting large- scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tun- ing performs comparably with conventional full-model tuning when downstream data are sufficient, whereas it is much worse under few- shot learning settings, which may hinder the application of prompt tuning. We attribute this low performance to the manner of initial- izing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to ob- tain a better initialization. We name this Pre- trained Prompt Tuning framework “PPT”. To ensure the generalization of PPT, we formu- late similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and effi- cient for using large-scale PLMs in practice. The code is publicly available at https:// github.com/thu-coai/PPT. 1 Introduction Fine-tuning pre-trained language models (PLMs) (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020) has made great progress in re- cent years. By tuning the entire model parameters, the versatile knowledge acquired from large-scale unlabeled corpora can be adapted to handling † Corresponding author. ∗ indicates equal contribution. various NLP tasks and outperform the approach of learning models from scratch (Han et al., 2021a). For simplicity, we name this full-model tuning as “FT”. As shown in Figure 1 (b) and (c), there are two mainstream FT approaches. The first one is task-oriented fine-tuning, where a task-specific head is added on top of PLMs, and the entire model is then fine-tuned by optimizing task-specific objectives on corresponding training data. The second one is prompt-oriented fine- tuning (Schick and Schütze, 2021a), which is inspired by the recent works utilizing language prompts to probe the knowledge in PLMs (Petroni et al., 2019; Brown et al., 2020). In prompt- oriented fine-tuning, data samples are converted to sequences containing prompt tokens, and down- stream tasks are formalized as language modeling problems. As shown in Figure 1 (c), by adding …

Content truncated to 500 words