Parameter-Efficient Fine-Tuning without Introducing New Latency

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4242–4260 July 9-14, 2023 ©2023 Association for Computational Linguistics Parameter-Efficient Fine-Tuning without Introducing New Latency Baohao Liao, Yan Meng, Christof Monz Language Technology Lab, University of Amsterdam {b.liao, y.meng, c.monz}@uva.nl Abstract Parameter-efficient fine-tuning (PEFT) of pre- trained language models has recently demon- strated remarkable achievements, effectively matching the performance of full fine-tuning while utilizing significantly fewer trainable pa- rameters, and consequently addressing the stor- age and communication constraints. Nonethe- less, various PEFT methods are limited by their inherent characteristics. In the case of sparse fine-tuning, which involves modifying only a small subset of the existing parameters, the se- lection of fine-tuned parameters is task- and domain-specific, making it unsuitable for feder- ated learning. On the other hand, PEFT meth- ods with adding new parameters typically in- troduce additional inference latency. In this paper, we demonstrate the feasibility of gen- erating a sparse mask in a task-agnostic man- ner, wherein all downstream tasks share a com- mon mask. Our approach, which relies solely on the magnitude information of pre-trained parameters, surpasses existing methodologies by a significant margin when evaluated on the GLUE benchmark. Additionally, we introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation, thereby achieving identical inference speed to that of full fine- tuning. Through extensive experiments, our proposed method attains a new state-of-the-art outcome in terms of both performance and stor- age efficiency, storing only 0.03% parameters of full fine-tuning.1 1 Introduction Pre-trained language models (PLMs) have served as a cornerstone for various natural language pro- cessing applications, favoring downstream tasks by offering a robust initialization (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019, 2020; Brown et al., 2020; Liao et al., 2022). Starting 1Code at https://github.com/baohaoliao/pafi_hiwi 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Fine-tuned Parameters (%) 82 83 84 85 86 87 88 VariousGLUE score (%) HiWi HiWi Full FT BitFit Prefix Tuning Adapter Pfeiffer Adapter LoRA MAM Adapter our PaFi our HiWi Figure 1: Comparison among various PEFT methods. (Zoom in for a better experience.) Marker size denotes the storage requirement. HiWi stores the least and its storage is invariant to the number of trainable parame- ters. Specific values are shown in Table 2. with a pre-trained checkpoint, a model can achieve significantly better performance on tasks of inter- est than the one from scratch. The most histori- cally common way to adapt PLMs to downstream tasks is to update all pre-trained parameters, full fine-tuning. While full fine-tuning produces nu- merous state-of-the-art results, it is impractical for storage-constrained and communication-frequent cases, like federated learning (McMahan et al., 2017), since it requires a full copy of the fine-tuned model for each task. This issue becomes more severe when PLMs are large-scale (Brown et al., 2020; Zhang et al., 2022; Hoffmann et al., 2022; Raffel et al., 2020; Scao et al., 2022; Touvron et al., 2023), the number …

Content truncated to 500 words