Revisiting Parameter-Efficient Tuning: Are We Really There Yet?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2612 - 2626 December 7-11, 2022 ©2022 Association for Computational Linguistics Revisiting Parameter-Efficient Tuning: Are We Really There Yet? Guanzheng Chen1, Fangyu Liu2, Zaiqiao Meng3, Shangsong Liang1,4,∗ 1Sun Yat-sen University 2University of Cambridge 3University of Glasgow 4Mohamed bin Zayed University of Artificial Intelligence guanzzh.chen@gmail.com, fl399@cam.ac.uk zaiqiao.meng@glasgow.ac.uk, liangshangsong@gmail.com Abstract Parameter-Efficient Tuning (PETuning) meth- ods have been deemed by many as the new paradigm for using pretrained language mod- els (PLMs). By tuning just a fraction amount of parameters comparing to full model finetun- ing, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investiga- tion into the training and evaluation of them. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competi- tive performance while finetuning remains to be the best-performing method in medium- and high-resource settings. We delve deeper into the cause of the instability and observed that the number of trainable parameters and training iterations are two main factors: reducing train- able parameters and prolonging training itera- tions may lead to higher stability in PETuning methods.1 1 Introduction Pretrained Language Models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have orchestrated tremendous progress in NLP in the past few years, achieving state of the art on a large variety of benchmarks such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019). Most successful applications of PLMs follow the pretraining-and-finetuning transfer learn- ing paradigm (Devlin et al., 2019), where PLMs are used as backbones to be combined with additional parameters and finetuned on downstream tasks in ∗Corresponding author. 1Our code is available at https://github.com/ guanzhchen/PETuning. 250 400 554 2500 3700 6000 7000 9427 67k 105k 364k 393k −20 −10 0 10 FT Adapter PT LoRA BitFit 0.6 1 1.4 Param.(%) Data Size Relative Performance Low-Resource (≤1k)Medium-Resource (1k~10k) High-Resource (≥10k) Figure 1: The relative performance difference of PETun- ing methods, i.e., Adapter, prefix tuning (PT), LoRA, BitFit, comparing with the full finetuning (FT) over different training data size of 12 tasks from GLUE and SuperGLUE. The tasks and their split into the three resource bands are illustrated in Appx. §B.1. The size of each point denotes the standard deviation and the colours of PETuning methods denote the percent- age of trainable parameters over different tasks com- pared to full finetuning. The key takeaway message is that PETuning methods outperform finetuning only in the low-resource tasks but remain on par or behind in medium and high-resource settings. an end-to-end manner. Whilst being simple and ef- fective, such paradigm requires task-specific tuning of the full model that consists of hundreds of mil- lions (Devlin et al., 2019; Liu …

Content truncated to 500 words