Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2226–2241, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2226 Masking as an Efficient Alternative to Finetuning for Pretrained Language Models Mengjie Zhao†*, Tao Lin‡*, Fei Mi‡, Martin Jaggi‡, Hinrich Schütze† † LMU Munich, Germany ‡ EPFL, Switzerland mzhao@cis.lmu.de, {tao.lin, fei.mi, martin.jaggi}@epfl.ch Abstract We present an efficient method of utilizing pre- trained language models, where we learn selec- tive binary masks for pretrained weights in lieu of modifying them through finetuning. Exten- sive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our masking scheme yields perfor- mance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred. Intrinsic evaluations show that representations computed by our bi- nary masked language models encode informa- tion necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accu- racy. This confirms that masking can be uti- lized as an efficient alternative to finetuning. 1 Introduction Finetuning a large pretrained language model like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), and XLNet (Yang et al., 2019) often yields competitive or even state-of-the-art results on NLP benchmarks (Wang et al., 2018, 2019). Given an NLP task, standard finetuning stacks a linear layer on top of the pretrained language model and then updates all parameters using mini-batch SGD. Vari- ous aspects like brittleness (Dodge et al., 2020) and adaptiveness (Peters et al., 2019) of this two-stage transfer learning NLP paradigm (Dai and Le, 2015; Howard and Ruder, 2018) have been studied. Despite the simplicity and impressive perfor- mance of finetuning, the prohibitively large number of parameters to be finetuned, e.g., 340 million in BERT-large, is a major obstacle to wider deploy- ment of these models. The large memory foot- print of finetuned models becomes more prominent * Equal contribution. when multiple tasks need to be solved – several copies of the millions of finetuned parameters have to be saved for inference. Recent work (Gaier and Ha, 2019; Zhou et al., 2019) points out the potential of searching neural architectures within a fixed model, as an alternative to optimizing the model weights for downstream tasks. Inspired by these results, we present mask- ing, a simple yet efficient scheme for utilizing pre- trained language models. Instead of directly updat- ing the pretrained parameters, we propose to select weights important to downstream NLP tasks while discarding irrelevant ones. The selection mecha- nism consists of a set of binary masks, one learned per downstream task through end-to-end training. We show that masking, when being applied to pretrained language models like BERT, RoBERTa, and DistilBERT (Sanh et al., 2019), achieves per- formance comparable to finetuning in tasks like part-of-speech tagging, named-entity recognition, sequence classification, and reading comprehen- sion. This is surprising in that a simple subselec- tion mechanism that does …

Content truncated to 500 words