One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7564–7580 July 9-14, 2023 ©2023 Association for Computational Linguistics One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning Guangtao Zeng∗ Peiyuan Zhang∗ Wei Lu StatNLP Research Group Singapore University of Technology and Design guangtao_zeng@mymail.sutd.edu.sg, {peiyuan_zhang, luwei}@sutd.edu.sg Abstract Fine-tuning pre-trained language models for multiple tasks tends to be expensive in terms of storage. To mitigate this, parameter-efficient transfer learning (PETL) methods have been proposed to address this issue, but they still re- quire a significant number of parameters and storage when being applied to broader ranges of tasks. To achieve even greater storage re- duction, we propose PROPETL, a novel method that enables efficient sharing of a single PETL module which we call prototype network (e.g., adapter, LoRA, and prefix-tuning) across layers and tasks. We then learn binary masks to select different sub-networks from the shared proto- type network and apply them as PETL modules into different layers. We find that the binary masks can determine crucial information from the network, which is often ignored in previous studies. Our work can also be seen as a type of pruning method, where we find that overparam- eterization also exists in the seemingly small PETL modules. We evaluate PROPETL on var- ious downstream tasks and show that it can outperform other PETL methods with approxi- mately 10% of the parameter storage required by the latter.1 1 Introduction With the release and wide application of numerous pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019), pre-training and subse- quently fine-tuning them becomes prevalent in nat- ural language processing (NLP). This yields good performance in many downstream tasks. However, such a paradigm requires the entire model to be up- dated and saved after fine-tuning. As PLMs grow in size, traditional fine-tuning becomes costly in stor- age, limiting the application to multi-task scenarios. In order to ameliorate this issue, many Parameter- Efficient Transfer Learning (PETL) methods have *The first two authors contributed equally. 1Our code is available at https://github.com/ ChaosCodes/ProPETL. Attention Feed Forward Attention Feed Forward Attention Feed Forward … … Shared Prototype Mask1 Shared Prototype Maskl Shared Prototype MaskN Figure 1: An illustration of our PROPETLAdapter model. Note that PROPETL is orthogonal to the specific PETL architectures. LoRA and prefix-tuning are also imple- mented in our framework. been proposed (Houlsby et al., 2019; Li and Liang, 2021; Hu et al., 2022). Rather than fine-tuning the entire model, they introduce new parameters and only fine-tune those additional parameters on downstream tasks, which drastically reduces the storage of the parameters required by each task. However, they still require a significant number of parameters when more tasks are considered. In this paper, we continue this line of research and target using even less storage per task. We observe that recent advancements in the field of PETL focus on finding better ways to apply the additional parameters, such as the adapter module after each feed-forward layer (Houlsby et al., …

Content truncated to 500 words