UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6253 - 6264 May 22-27, 2022 c⃝2022 Association for Computational Linguistics UNIPELT: A Unified Framework for Parameter-Efficient Language Model Tuning Yuning Mao1∗, Lambert Mathias2, Rui Hou2, Amjad Almahairi2, Hao Ma2, Jiawei Han1, Wen-tau Yih2, Madian Khabsa2 1University of Illinois Urbana-Champaign {yuningm2, hanj}@illinois.edu 2Meta AI {mathiasl, rayhou, aalmah, haom, scottyih, mkhabsa}@fb.com Abstract Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, different PELT methods may perform rather differently on the same task, making it non- trivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks. In light of model diversity and the difficulty of model selection, we propose a unified framework, UNIPELT, which incorpo- rates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mech- anism. On the GLUE benchmark, UNIPELT consistently achieves 1~4% gains compared to the best individual PELT method that it incor- porates and outperforms fine-tuning under dif- ferent setups. Moreover, UNIPELT generally surpasses the upper bound that takes the best performance of all its submodules used indi- vidually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods.1 1 Introduction As pre-trained language models (PLMs) (Devlin et al., 2019) grow larger and larger (Brown et al., 2020), it becomes increasingly infeasible to per- form conventional fine-tuning, where separate repli- cas of the model parameters are modified per single task. To solve the issue, there has recently been a surge of studies on parameter-efficient language model tuning (PELT), namely how to effectively tune the PLMs with fewer trainable parameters. Existing PELT research generally aims at achiev- ing performance comparable to fine-tuning with ∗Work was done during internship at Meta AI. 1Our code can be found at https://github.com/ morningmoni/UniPELT. Adapter Multi-Head Attention Add & Norm Feedforward Add & Norm Add & Norm + Prefix-tuning hin hFN hA hF LoRA + + Q K V WQ WV WK WDown WUp PK PV WUp WUp WDown WDown GL GA GP Figure 1: Illustration of UNIPELT, which subsumes existing PELT methods as submodules and controls them via gating mechanism G. Different (combinations of) submodules can be activated for different samples. The trainable parameters are shown in blue. as few trainable parameters as possible, which has seen significant progress – the task-specific train- able parameters used in most recent approaches (Lester et al., 2021; Guo et al., 2021) are almost negligible compared to the total parameters of the PLM (<1%). A more challenging yet less studied problem is whether one can achieve better perfor- mance than fine-tuning with fewer parameters. Re- cent studies (He et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al., 2021b) find that some PELT methods are …

Content truncated to 500 words