Parameter-Efficient Tuning Makes a Good Classification Head

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7576 - 7586 December 7-11, 2022 ©2022 Association for Computational Linguistics Parameter-Efficient Tuning Makes a Good Classification Head Zhuoyi Yang†∗ Ming Ding†∗ Yanhui Guo‡ Qingsong Lv† Jie Tang† †Tsinghua University ‡Shandong Woman University {yangzy22,dm18}@mails.tsinghua.edu.cn jietang@tsinghua.edu.cn Abstract In recent years, pretrained models revolution- ized the paradigm of natural language under- standing (NLU), where we append a randomly initialized classification head after the pre- trained backbone, e.g. BERT, and finetune the whole model. As the pretrained backbone makes a major contribution to the improve- ment, we naturally expect a good pretrained classification head can also benefit the train- ing. However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining (LP- FT) ineffective. In this paper, we find that parameter-efficient tuning makes a good clas- sification head, with which we can simply re- place the randomly initialized heads for a stable performance gain. Our experiments demon- strate that the classification head jointly pre- trained with parameter-efficient tuning consis- tently improves the performance on 9 tasks in GLUE and SuperGLUE. 1 Introduction Fine-tuning is the most prevalent paradigm to lever- age pretrained language models for the best per- formance on specific tasks (Kenton and Toutanova, 2019). Usually, a task-oriented classification head is grafted onto the final layer of the pretrained back- bone, mostly Transformers (Vaswani et al., 2017), and then the whole model is trained for the down- stream task. Compared with training a large Trans- former from scratch, the pretrained backbone has already learned to extract features about grammar, semantics and high-level understanding, making it easy to adapt the model for NLP tasks. Although the good initial weights of the pre- trained backbone are the key factor of the effective- ness, the initial weights of the classification heads are, however, largely under-explored. To the best ∗Equal contribution. Codes are at https://github. com/THUDM/Efficient-Head-Finetuning. Figure 1: The amount of feature change after finetuning RoBERTa-large on RTE with different strategies, where “features” denote the final-layer output of the pretrained RoBERTa backbone, a.k.a. the input of the classification head. The feature change is measured by the L2 distance between the features on the training set before and after finetuning. of our knowledge, the usage of randomly initial- ized classification heads is still the overwhelmingly dominant choice in NLP. Recently, LP-FT (Kumar et al., 2022) finds that on many computer vision benchmarks, the performance can be promoted, especially for out-of-distribution (OOD) data, if we first only finetune the linear classification head (probe) with the pretrained backbone frozen, and then finetune the whole model. Does this kind of head-first finetuning technique also work in NLP? Furthermore, since the principle is basically to reduce the change of features during finetuning (Figure 1), is it possible to upgrade LP- FT to a general method to improve generalization, instead of only OOD setting? In this paper, we give a positive answer via parameter-efficient tuning. To …

Content truncated to 500 words