Prompt Tuning for Discriminative Pre-trained Language Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 3468 - 3473 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Prompt Tuning for Discriminative Pre-trained Language Models Yuan Yao1, Bowen Dong1, Ao Zhang2, Zhengyan Zhang1, Ruobing Xie3, Zhiyuan Liu1,4,5,6†, Leyu Lin3, Maosong Sun1,4,5,6†, Jianyong Wang1 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology 2Department of Computer Science, National University of Singapore, Singapore 3WeChat Search Application Department, Tencent, China 4Institute for Artificial Intelligence, Tsinghua University, Beijing, China 5Institute Guo Qiang, Tsinghua University, Beijing, China 6International Innovation Center of Tsinghua University, Shanghai, China yaoyuanthu@163.com dongbw18@mails.tsinghua.edu.cn Abstract Recent works have shown promising results of prompt tuning in stimulating pre-trained lan- guage models (PLMs) for natural language processing (NLP) tasks. However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs that are pre-trained to generate target tokens, such as BERT (Devlin et al., 2019). It is still unknown whether and how discriminative PLMs, e.g., ELECTRA (Clark et al., 2020), can be effec- tively prompt-tuned. In this work, we present DPT, the first prompt tuning framework for discriminative PLMs, which reformulates NLP tasks into a discriminative language model- ing problem. Comprehensive experiments on text classification and question answering show that, compared with vanilla fine-tuning, DPT achieves significantly higher performance, and also prevents the unstable problem in tuning large PLMs in both full-set and low-resource settings. The source code and experiment de- tails of this paper can be obtained from https: //github.com/thunlp/DPT. 1 Introduction Recent years have witnessed the great success of the pre-training-then-fine-tuning paradigm in natu- ral language processing (NLP) (Devlin et al., 2019; Yang et al., 2019; Clark et al., 2020; Lan et al., 2020; Raffel et al., 2020). Typically, language mod- els are first pre-trained on large-scale corpora via self-supervised generative or discriminative tasks to learn universal text representations, and then fine-tuned to adapt to downstream tasks (Qiu et al., 2020; Xu et al., 2021). However, the significant gap † Corresponding authors: Z.Liu (liuzy@tsinghua.edu.cn), M.Sun (sms@tsinghua.edu.cn) between the objective forms of model pre-training and fine-tuning hinders taking full advantage of PLMs in downstream tasks (Liu et al., 2021). Prompt tuning has recently shown its effective- ness in stimulating the capability of PLMs by trans- forming downstream tasks into the same form as pre-training (Petroni et al., 2019; Brown et al., 2020; Schick and Schütze, 2021; Gao et al., 2021; Liu et al., 2021). However, to the best of our knowl- edge, existing works focus on prompt-tuning gen- erative PLMs (i.e., PLMs pre-trained by generat- ing target textual tokens from the context, such as BERT (Devlin et al., 2019) and GPT (Brown et al., 2020)). It is still unknown whether and how dis- criminative PLMs can be effectively prompt-tuned (i.e., PLMs pre-trained by discriminating replaced tokens, such as ELECTRA (Clark et al., 2020) and WKLM (Xiong et al., 2020)). Since discriminative PLMs typically enjoy competitive performance and superior computational efficiency compared with their generative counterparts …

Content truncated to 500 words