AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5744 - 5760 December 7-11, 2022 ©2022 Association for Computational Linguistics AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning Yaqing Wang∗ Purdue University wang5075@purdue.edu Sahaj Agarwal Microsoft sahagar@microsoft.com Subhabrata Mukherjee† Microsoft Research submukhe@microsoft.com Xiaodong Liu Microsoft Research Jing Gao Purdue University Ahmed Hassan Awadallah Microsoft Research Jianfeng Gao Microsoft Research Abstract Standard fine-tuning of large pre-trained lan- guage models (PLMs) for downstream tasks requires updating hundreds of millions to bil- lions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules – given the underlying PEFT method of choice – intro- duced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby (Houlsby et al., 2019) or a mix- ture of low rank decomposition matrices like LoRA (Hu et al., 2021) to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1 − 0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks. Code and models are made available at https://aka.ms/AdaMix. 1 Introduction Standard fine-tuning of large pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020; Raffel et al., 2019) to downstream tasks requires updating all model pa- rameters. Given the ever-increasing size of PLMs (e.g., 175 billion parameters for GPT-3 (Brown et al., 2020) and 530 billion parameters for MT- NLG (Smith et al., 2022)), even the fine-tuning step becomes expensive as it requires storing a full copy ∗Work done at Microsoft Research. †Corresponding author and project lead. 0 0.5 1 1.5 2 Fine-tuned Parameters (%) 85 86 87 88 89 90 Average Score Pfeiffer 87.9 Houlsby 86.4 LoRA 88.6 AdaMix 89.9 (ours) Pfeiffer 88.4 Houlsby 87.8 Full Fine-tuning 88.9 Figure 1: Performance of different parameter-efficient fine-tuning methods on GLUE development set with RoBERTa-large encoder following a setup similar to (Houlsby et al., 2019) for fair comparison. We re- port the performance of Pfeiffer (Pfeiffer et al., 2021), Houlsby (Houlsby et al., 2019) and LoRA (Hu et al., 2021) with their default number of fine-tuned parame- ters as well as the number of fine-tuned parameters used in AdaMix with a mixture of adaptations . Red dash shows the performance of full model fine-tuning. of model weights for every task. To address these challenges, recent works have developed parameter- efficient fine-tuning (PEFT) techniques. These approaches typically underperform standard full model …

Content truncated to 500 words