Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11156–11172 July 9-14, 2023 ©2023 Association for Computational Linguistics Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning Tianxiang Sun∗ Zhengfu He∗ Qin Zhu Xipeng Qiu† Xuanjing Huang School of Computer Science, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {txsun19,zfhe19,xpqiu,xjhuang}@fudan.edu.cn zhuq22@m.fudan.edu.cn Abstract Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. Although prompt tuning has been shown to match the performance of full model tuning when training data is sufficient, it tends to struggle in few-shot learning settings. In this paper, we present Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tun- ing for few-shot learning. MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks. On downstream tasks, the pre-trained prompts are selectively activated and combined, leading to strong compositional generalization to unseen tasks. To bridge the gap between pre-training and fine-tuning, we formulate up- stream and downstream tasks into a unified ma- chine reading comprehension task. Extensive experiments under two learning paradigms, i.e., gradient descent and black-box tuning, show that MP2 significantly outperforms prompt tun- ing, full model tuning, and prior prompt pre- training methods in few-shot settings. In ad- dition, we demonstrate that MP2 can achieve surprisingly fast and strong adaptation to down- stream tasks by merely learning 8 parameters to combine the pre-trained modular prompts. 1 Introduction Pre-trained models (PTMs) (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Qiu et al., 2020) with prompt-based learning have achieved remarkable progress in few-shot learning. A ma- jor reason behind their success is the closed gap between upstream pre-training and downstream fine-tuning (Liu et al., 2021a; Sun et al., 2022b). Since the downstream tasks are reformulated into a unified (masked) language modeling ((M)LM for short) task, one can reuse the pre-trained (M)LM head instead of training a randomly initialized clas- sification head to solve tasks with limited data. ∗ Equal contribution. † Corresponding author. Figure 1: MP2 achieves fast adaptation to downstream tasks through three steps: (1) Self-supervised pre- training on large-scale unlabeled data. (2) Pre-training modular prompts and the corresponding router with multi-task learning. (3) A subset of prompts is activated and tuned for adaptation to downstream tasks. However, prompt-based learning (e.g., PET (Schick and Schütze, 2021) and LM-BFF (Gao et al., 2021)) usually fine-tunes all the parameters of the PTM for each downstream task, which can be computa- tionally expensive and deployment-inefficient, es- pecially for large PTMs such as GPT-3 (Brown et al., 2020). Recently, much effort has been devoted to parameter-efficient prompt tuning (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021c; Sun et al., 2022c), which only learns a small number of soft prompt parameters while keeping the main body of the PTM untouched. In contrast to full model tuning, prompt tuning can get specialized models for specific tasks by simply attaching task- specific prompts, and therefore is highly efficient …

Content truncated to 500 words