XPrompt: Exploring the Extreme of Prompt Tuning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11033 - 11047 December 7-11, 2022 ©2022 Association for Computational Linguistics XPROMPT: Exploring the Extreme of Prompt Tuning Fang Ma , Chen Zhang , Lei Ren , Jingang Wang , Qifan Wang , Wei Wu , Xiaojun Quan , Dawei Song Beijing Institute of Technology {mfang,czhang,dwsong}@bit.edu.cn Meituan {wangjingang02,wuwei30}@meituan.com, renlei_work@163.com Meta AI wqfcr@fb.com Sun Yat-Sen University quanxj3@mail.sysu.edu.cn Abstract Prompt tuning learns soft prompts to condi- tion the frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner. While prompt tuning has gradually reached the performance level of fine-tuning as the model scale increases, there is still a large performance gap between prompt tuning and fine-tuning for models of moderate and small scales (typically less than 11B parameters). In this paper, we empirically show that the trained prompt tokens can have a negative impact on a downstream task and thus degrade its performance. To bridge the gap, we propose a novel PROMPT tuning model with an eXtremely small scale (XPROMPT) un- der the regime of lottery tickets hypothesis. Specifically, XPROMPT eliminates the negative prompt tokens at different granularity levels through a hierarchical structured pruning, yield- ing a more parameter-efficient prompt yet with a competitive performance. Comprehensive ex- periments are carried out on the SuperGLUE tasks, and the results indicate that XPROMPT is able to close the performance gap at smaller model scales. 1 1 Introduction Pre-trained Language Models (PLMs) have been widely applied and achieved a remarkable suc- cess in various NLP tasks (Devlin et al., 2019; Raffel et al., 2020; Zhou et al., 2020) under the pretrain-then-finetune paradigm (Liu et al., 2019). Despite of its compelling performance, fine-tuning is parameter-inefficient for large scale PLMs due to the fact that the memory footprint is proportional to the number of trainable parameters whose gra- dients and optimizer states need to be stored (Guo et al., 2021). authors.Dawei Song and Jingang Wang are the corresponding 1The code is available at https://github.com/BD-MF/ XPrompt. 770M(T5-Large) 3B(T5-XL) 11B(T5-XXL) Model Scales 75 80 85 90 95 SuperGLUE Score Fine-Tuning Prompt-Tuning XPrompt(Ours) Figure 1: XPROMPT outperforms the vanilla Prompt- Tuning (Lester et al., 2021) and can significantly im- prove over Prompt-Tuning across tasks and model scales. It is worth noting that there is a small perfor- mance gap between prompt tuning and fine-tuning on T5-XXL (11B) due to different hyperparameter settings and initialization. Similar observations have been found in Figure3-a and Figure3-b of Lester et al. (2021). Recently, Prompt-Tuning (Lester et al., 2021; Liu et al., 2021b) has been proposed to address this issue by prepending a soft prompt to the in- put and only updating the parameters of prompt tokens during tuning. Prompt-Tuning provides a parameter-efficient alternative to fine-tuning, since the scale of the soft prompt is tens of thousand smaller. It is also conceptually simpler and more flexible than other parameter-efficient tuning meth- ods (such as Adapters), that require intrusive mod- ifications to transformer layers (Houlsby et al., 2019; Guo et …

Content truncated to 500 words