Parameter-Efficient Transfer Learning with Diff Pruning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4884–4896 August 1–6, 2021. ©2021 Association for Computational Linguistics 4884 Parameter-Efficient Transfer Learning with Diff Pruning Demi Guo Harvard University dguo@college.harvard.edu Alexander M. Rush Cornell University arush@cornell.edu Yoon Kim MIT CSAIL MIT-IBM Watson AI yoonkim@mit.edu Abstract The large size of pretrained networks makes them difficult to deploy for multiple tasks in storage-constrained settings. Diff pruning en- ables parameter-efficient transfer learning that scales well with new tasks. The approach learns a task-specific “diff” vector that ex- tends the original pretrained parameters. This diff vector is adaptively pruned during train- ing with a differentiable approximation to the L0-norm penalty to encourage sparsity. As the number of tasks increases, diff pruning re- mains parameter-efficient, as it requires stor- ing only a small diff vector for each task. Since it does not require access to all tasks dur- ing training, it is attractive in on-device de- ployment settings where tasks arrive in stream or even from different providers. Diff prun- ing can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s pa- rameters per task and scales favorably in com- parison to popular pruning approaches. 1 Introduction Task-specific finetuning of pretrained deep net- works is the dominant paradigm in contemporary NLP, achieving state-of-the-art results across a suite of natural language understanding tasks (De- vlin et al., 2019; Liu et al., 2019c; Yang et al., 2019; Lan et al., 2020). While straightforward and em- pirically effective, this approach is difficult to scale to multi-task, memory-constrained settings (e.g. for on-device applications), as it requires shipping and storing a full set of model parameters for each task. Inasmuch as these models are learning gen- eralizable, task-agnostic language representations through self-supervised pretraining, finetuning the entire model for each task seems especially profli- gate. Code: https://github.com/dguo98/DiffPruning A popular approach to parameter-efficiency is to learn smaller compressed models for each task (Gordon et al., 2020; Sajjad et al., 2020; Zhao et al., 2020; Sanh et al., 2020). Such approaches face a steep sparsity/performance tradeoff and keep a substantial amount of nonzero parameters per task (e.g. 10%-30%). Multi-task learning and feature- based transfer allow for more parameter-efficient transfer learning per task (Liu et al., 2019b; Clark et al., 2019; Stickland & Murray, 2019; Reimers & Gurevych, 2019). These methods train a small num- ber of additional parameters (e.g. a linear layer) on top of a shared model. However, multi-task learn- ing generally requires access to all tasks during training to prevent catastrophic forgetting (French, 1999), while feature-based transfer learning (e.g. based on task-agnostic sentence representations) is typically outperformed by finetuning (Howard & Ruder, 2018). An appealing middle ground is to finetune an extension of the base model for specific tasks. This approach captures the training benefits of fine- tuning while maintaining the task modularity of feature-based transfer. For example, Adapters (Re- buffi et al., 2018) use smaller, task-specific modules …

Content truncated to 500 words