An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10617 - 10625 December 7-11, 2022 ©2022 Association for Computational Linguistics An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning Mohammad AkbarTajari∗ 1, Sara Rajaee∗ 2, and Mohammad Taher Pilehvar3 1 Sharif University of Technology, Iran 2 University of Amsterdam, Netherlands 3 Tehran Institute for Advanced Studies, Khatam University, Iran m.akbarTajari@gmail.com s.rajaee@uva.nl mp792@cam.ac.uk Abstract Parameter-efficient fine-tuning approaches have recently garnered a lot of attention. Having considerably lower number of train- able weights, these methods can bring about scalability and computational effectiveness. In this paper, we look for optimal sub- networks and investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. Our empirical results suggest that every transformer module in BERT can act as a winning ticket: fine-tuning each specific module while keeping the rest of the network frozen can lead to comparable performance to the full fine-tuning. Among different modules, LayerNorms exhibit the best capacity for knowledge transfer with limited trainable weights, to the extent that, with only 0.003% of all parameters in the layer-wise analysis, they show acceptable performance on various target tasks. On the reasons behind their effectiveness, we argue that their notable performance could be attributed to their high-magnitude weights compared to that of the other modules in the pre-trained BERT. The code for this paper is freely available at https://github.com/ m-tajari/transformer-transferability. 1 Introduction Fine-tuning is widely used as a procedure to em- ploy the knowledge learned during pre-training of language models for specific tasks (Howard and Ruder, 2018; Peters et al., 2019; Merchant et al., 2020; Zhou and Srikumar, 2022). However, fine- tuning can be a computationally expensive pro- cess, given that it usually involves updating all the parameters in transformer-based models which are often massive in size. Parameter-efficient fine- tuning methods try to ameliorate this by reducing the number of updatable parameters during fine- tuning. ∗The authors contributed equally to this work. Adapters (Houlsby et al., 2019; Pfeiffer et al., 2020; Wang et al., 2021; Rücklé et al., 2021; Karimi Mahabadi et al., 2021; Hu et al., 2021) try to circumvent this issue by inserting light- weight modules in the transformer blocks, tun- ing of which usually results in comparable per- formance to the full fine-tuning (while the num- ber of updatable parameters is significantly lower). Nevertheless, introducing new parameters to an already-large model can be considered a draw- back. Another category of parameter-efficient fine-tuning methods is based on the Lottery Ticket Hypothesis (Prasanna et al., 2020), where the goal is to find a small subset of parameters that can compete with the full fine-tuning setting. Vari- ous subsets of network parameters have been sug- gested as the winning ticket, including the con- nections with high magnitudes (Han et al., 2015), identity mappings (Lin et al., 2020), and dominant dimensions (Guo et al., 2021). In this paper, we study the ability of differ- ent modules of a transformer block in …

Content truncated to 500 words