When BERT Plays the Lottery, All Tickets Are Winning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3208–3229, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3208 When BERT Plays the Lottery, All Tickets Are Winning Sai Prasanna∗ Zoho Labs Zoho Corporation Chennai, India saiprasanna.r@zohocorp.com Anna Rogers∗ Center for Social Data Science Copenhagen University Copenhagen, Denmark arogers@sodas.ku.dk Anna Rumshisky Dept. of Computer Science Univ. of Massachusetts Lowell Lowell, USA arum@cs.uml.edu Abstract Large Transformer-based models were shown to be reducible to a smaller number of self- attention heads and layers. We consider this phenomenon from the perspective of the lot- tery ticket hypothesis, using both structured and magnitude pruning. For fine-tuned BERT, we show that (a) it is possible to find sub- networks achieving performance that is com- parable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. Strikingly, with structured pruning even the worst possi- ble subnetworks remain highly trainable, indi- cating that most pre-trained BERT weights are potentially useful. We also study the “good” subnetworks to see if their success can be at- tributed to superior linguistic knowledge, but find them unstable, and not explained by mean- ingful self-attention patterns. 1 Introduction Much of the recent progress in NLP is due to the transfer learning paradigm in which Transformer- based models first try to learn task-independent linguistic knowledge from large corpora, and then get fine-tuned on small datasets for specific tasks. However, these models are overparametrized: we now know that most Transformer heads and even layers can be pruned without significant loss in performance (Voita et al., 2019; Kovaleva et al., 2019; Michel et al., 2019). One of the most famous Transformer-based mod- els is BERT (Devlin et al., 2019). It became a must-have baseline and inspired dozens of studies probing it for various kinds of linguistic informa- tion (Rogers et al., 2020b). We conduct a systematic case study of fine- tuning BERT on GLUE tasks (Wang et al., 2018) from the perspective of the lottery ticket hypothesis ∗Equal contribution (Frankle and Carbin, 2019). We experiment with and compare magnitude-based weight pruning and importance-based pruning of BERT self-attention heads (Michel et al., 2019), which we extend to multi-layer perceptrons (MLPs) in BERT. With both techniques, we find the “good” subnet- works that achieve 90% of full model performance, and perform considerably better than similarly- sized subnetworks sampled from other parts of the model. However, in many cases even the “bad” subnetworks can be re-initialized to the pre-trained BERT weights and fine-tuned separately to achieve strong performance. We also find that the “good” networks are unstable across random initializations at fine-tuning, and their self-attention heads do not necessarily encode meaningful linguistic patterns. 2 Related Work Multiple studies of BERT concluded that it is con- siderably overparametrized. In particular, it is pos- sible to ablate elements of its architecture without loss in performance or even with slight gains (Ko- valeva et al., 2019; Michel et al., 2019; Voita et al., 2019). This explains the success …

Content truncated to 500 words