Finding the Dominant Winning Ticket in Pre-Trained Language Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 1459 - 1472 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Finding the Dominant Winning Ticket in Pre-Trained Language Models Zhuocheng Gong1, Di He2, Yelong Shen3, Tie-yan Liu2, Weizhu Chen3, Dongyan Zhao1,4,5∗, Ji-rong Wen6 and Rui Yan6∗ 1Wangxuan Institute of Computer Technology, Peking University, China 2Microsoft Research, Beijing, China, 3Microsoft Azure AI 4Artificial Intelligence Institute of Peking University 5State Key Laboratory of Media Convergence Production 6Gaoling School of Artificial Intelligence, Renmin University of China {gzhch, zhaody}@pku.edu.cn, {jrwen, ruiyan}@ruc.edu.cn {dihe, yelong.shen, tyliu, wzchen}@microsoft.com Abstract The Lottery Ticket Hypothesis suggests that for any over-parameterized model, a small subnet- work exists to achieve competitive performance compared to the backbone architecture. In this paper, we study whether there is a winning lottery ticket for pre-trained language models, which allow the practitioners to fine-tune the parameters in the ticket but achieve good down- stream performance. To achieve this, we regu- larize the fine-tuning process with L1 distance and explore the subnetwork structure (what we refer to as the "dominant winning ticket"). Em- pirically, we show that (a) the dominant win- ning ticket can achieve performance that is com- parable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant win- ning ticket has a natural structure within each parameter matrix. Strikingly, we find that a dominant winning ticket that takes up 0.05% of the parameters can already achieve satisfac- tory performance, indicating that the PLM is significantly reducible during fine-tuning. 1 Introduction Pre-trained Language Models (PLMs) have shown significant performance on various natural lan- guage processing (NLP) tasks (Devlin et al., 2018; Liu et al., 2019). However, as the number of model parameters gets huge, fine-tuning such models be- comes inefficient. Many previous works target parameter-efficient fine-tuning approaches by freez- ing the PLM parameters. One can either freeze a subset of the parameters (Zaken et al., 2021) and *corresponding authors: Dongyan Zhao (zhaody@pku.edu.cn) and Rui Yan (ruiyan@ruc.edu.cn) Figure 1: An illustration of the structure of the dominant winning ticket. fine-tune the remainings or freeze all of them and plug in light modules with new learnable parame- ters (Houlsby et al., 2019; Mahabadi et al., 2021; Hu et al., 2021). In parallel to this line of study, an emerging sub- field has explored the possibility of training smaller subnetworks in place of the full models without hurting performance (Lee et al., 2018; Wang et al., 2020). Among them, the lottery ticket hypothe- sis (LTH) (Frankle and Carbin, 2018) has attracted much attention. LTH demonstrates that an over- parameterized network contains "winning tickets" (small-scale subnetworks) that can 1) match the performance of the full model; and 2) outper- form randomly sampled subnetworks of the same size. Winning tickets have been verified to exist in PLMs (Prasanna et al., 2020). In this paper, we provide an interesting result by showing that a subnetwork, which we refer to as the dominant winning ticket, exists in the PLM. It can …

Content truncated to 500 words