ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6655 - 6672 December 7-11, 2022 ©2022 Association for Computational Linguistics ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts Akari Asai♡ Mohammadreza Salehi♡ Matthew E. Peters♢ Hannaneh Hajishirzi♡♢ ♡ University of Washington ♢ Allen Institute for AI {akari, mrsalehi, hannaneh}@cs.washington.edu matthewp@allenai.org Abstract This work introduces a new multi-task, parameter-efficient language model (LM) tun- ing method that learns to transfer knowledge across different tasks via a mixture of soft prompts—small prefix embedding vectors pre- trained for different tasks. Our method, called ATTEMPT (ATTEntional Mixtures of Prompt Tuning), obtains source prompts as encodings of large-scale source tasks into a small number of parameters and trains an attention module to interpolate the source prompts and a newly initialized target prompt for every instance in the target task. During training, only the target task prompt and the attention weights, which are shared between tasks in multi-task train- ing, are updated, while the original LM and source prompts are intact. ATTEMPT is highly parameter-efficient (e.g., updates 2,300 times fewer parameters than full fine-tuning), while achieving high task performance using knowl- edge from high-resource tasks. Moreover, it is modular using pre-trained soft prompts and can flexibly add or remove source prompts for effec- tive knowledge transfer. Our experimental re- sults across 21 diverse NLP datasets show that ATTEMPT significantly outperforms prompt tuning and outperforms or matches fully fine- tuned or other parameter-efficient tuning ap- proaches that use over ten times more parame- ters. Finally, ATTEMPT outperforms previous work in few-shot learning settings.1 1 Introduction Fine-tuning all the parameters of large language models (LMs) given target task training data is the most common practice for optimizing task perfor- mance (Devlin et al., 2019; Raffel et al., 2020). A recent line of research introduces parameter- efficient tuning methods (Houlsby et al., 2019; Li and Liang, 2021; Ben Zaken et al., 2022) that only 1Our code is available at https://github.com/ AkariAsai/ATTEMPT. Frozen LM 🧊 X y Pmnli Psst-2 Psquad Ptarget Pinstance Source prompts 🧊 Target prompt Attention Module 🔥 Figure 1: ATTEMPT combines multiple soft prompts trained on large-scale datasets (source prompts) to gen- erate instance-wise prompts for a target task. At target task training, the LM and the source prompts are intact. update a small number of LM parameters; how- ever, increasing efficiency often decreases the task performance (He et al., 2022). Moreover, these models are trained only using the task training data and do not benefit from large collection of other NLP tasks (Liu et al., 2019a). We posit that parameter-efficient tuning methods can leverage rich knowledge of high-resource tasks to improve both training efficiency and task performance. This work introduces a new parameter-efficient, modular multi-task tuning method called ATTEMPT (ATTEntional Mixtures of Prompt Tuning, pre- viewed in Figure 1). ATTEMPT efficiently inte- grates knowledge from multiple tasks via a mix- ture of trainable soft prompts preprended to the in- put, keeping the original LM completely frozen. It first pre-trains transferable soft …

Content truncated to 500 words