Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6524–6538 August 1–6, 2021. ©2021 Association for Computational Linguistics 6524 Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization Chen Liang∗ 1, Simiao Zuo1, Minshuo Chen1, Haoming Jiang1, Xiaodong Liu2, Pengcheng He3, Tuo Zhao1, Weizhu Chen3 1 Georgia Institute of Technology, 2 Microsoft Research, 3 Microsoft Azure AI {cliang73, simiaozuo, mchen393, jianghm, tourzhao}@gatech.edu {xiaodl,penhe,wzchen}@microsoft.com Abstract The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of “lottery tick- ets”, and training a certain collection of them (i.e., a subnetwork) can match the performance of the full model. In this paper, we study such a collec- tion of tickets, which is referred to as “winning tickets”, in extremely over-parametrized models, e.g., pre-trained language models. We observe that at certain compression ratios, the generalization performance of the winning tickets can not only match but also exceed that of the full model. In par- ticular, we observe a phase transition phenomenon: As the compression ratio increases, generalization performance of the winning tickets first improves then deteriorates after a certain threshold. We refer to the tickets on the threshold as “super tickets”. We further show that the phase transition is task and model dependent — as the model size becomes larger and the training data set becomes smaller, the transition becomes more pronounced. Our experi- ments on the GLUE benchmark show that the super tickets improve single task fine-tuning by 0.9 points on BERT-base and 1.0 points on BERT-large, in terms of task-average score. We also demonstrate that adaptively sharing the super tickets across tasks benefits multi-task learning1. 1 Introduction The Lottery Ticket Hypothesis (LTH, Frankle and Carbin (2018)) suggests that an over-parameterized network consists of “lottery tickets”, and training a certain collection of them (i.e., a subnetwork) can 1) match the performance of the full model; and 2) ∗Work was done at Microsoft Azure AI. 1Our codes are available at https://github.com/cliang1453/ super-structured-lottery-tickets. outperform randomly sampled subnetworks of the same size (i.e., “random tickets”). The existence of such a collection of tickets, which is usually referred to as “winning tickets”, indicates the po- tential of training a smaller network to achieve the full model’s performance. LTH has been widely explored in across various fields of deep learning (Frankle et al., 2019; Zhou et al., 2019; You et al., 2019; Brix et al., 2020; Movva and Zhao, 2020; Girish et al., 2020). Aside from training from scratch, such winning tickets have demonstrated their abilities to transfer across tasks and datasets (Morcos et al., 2019; Yu et al., 2019; Desai et al., 2019; Chen et al., 2020a). In natural language processing, Chen et al. (2020b); Prasanna et al. (2020) have shown existence of the winning tickets in pre-trained language models. These tickets can be identified when fine-tuning the pre-trained models on downstream tasks. As the pre-trained models are usually extremely over- parameterized (e.g., BERT Devlin et al. (2019), GPT-3 Brown …

Content truncated to 500 words