IDPG: An Instance-Dependent Prompt Generation Method

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5507 - 5521 July 10-15, 2022 ©2022 Association for Computational Linguistics IDPG: An Instance-Dependent Prompt Generation Method Zhuofeng Wu1∗ Sinong Wang2 Jiatao Gu2 Rui Hou2 Yuxiao Dong2,3† V.G.Vinod Vydiswaran4,1 Hao Ma2 1School of Information, University of Michigan 2Meta AI 3Tsinghua University 4Department of Learning Health Sciences, University of Michigan {zhuofeng, vgvinodv}@umich.edu {sinongwang, jgu, rayhou, haom}@fb.com yuxiaod@tsinghua.edu.cn Abstract Prompt tuning is a new, efficient NLP trans- fer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. It freezes the pre-trained lan- guage model and only optimizes a few task- specific prompts. In this paper, we propose a conditional prompt generation method to gen- erate prompts for each input instance, referred to as the Instance-Dependent Prompt Genera- tion (IDPG). Unlike traditional prompt tuning methods that use a fixed prompt, IDPG intro- duces a lightweight and trainable component to generate prompts based on each input sentence. Extensive experiments on ten natural language understanding (NLU) tasks show that the pro- posed strategy consistently outperforms vari- ous prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model pa- rameters.1 1 Introduction In recent years, pre-training a transformer model on a large corpus with language modeling tasks and fine-tuning it on different downstream tasks has be- come the primary transfer learning paradigm in nat- ural language processing (Devlin et al., 2019). No- tably, this paradigm requires updating and storing all the model parameters for each downstream task. As the model size proliferates (e.g., 330M param- eters for BERT (Devlin et al., 2019) and 175B for GPT-3 (Brown et al., 2020)), it becomes computa- tionally expensive and challenging to fine-tune the entire pre-trained language model (LM). Thus, it is natural to ask whether we can transfer the knowl- edge of a pre-trained LM to downstream tasks by keeping most of the parameters fixed and tuning only a small fraction of them. ∗ Work partially done while interning at Meta AI. † Work done when at Meta AI. 1Our code is publicly available at https://github. com/CSerxy/IDPG. 0.001 0.01 0.1 1 10 100 88 89 90 91 92 93 prompt tuning P-tuning v2 Compacter Adapter IDPG-PHM IDPG-PHM-GloVe RoBERTa-FT Percentage of trained parameters per task (relative to RoBERTa) Average Score Figure 1: Overall evaluation of competing approaches on ten NLU tasks, with parameters from classification heads excluded. Our method approaches RoBERTa- FT’s performance and uses fewer parameters than Adapter-based methods. Previous studies have attempted to address this question from different perspectives. One line of research (Li and Liang, 2021) suggests aug- menting the model with smaller, trainable mod- ules and freezing the original transformer weights. Adapters (Houlsby et al., 2019; Pfeiffer et al., 2021, 2020), for example, insert a small set of addi- tional modules between each transformer layer. Only these additional and task-specific modules are trained during fine-tuning, reducing the number …

Content truncated to 500 words