Should You Mask 15% in Masked Language Modeling?

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985–3000 May 2-6, 2023 ©2023 Association for Computational Linguistics Should You Mask 15% in Masked Language Modeling? Alexander Wettig∗ Tianyu Gao∗ Zexuan Zhong Danqi Chen Department of Computer Science, Princeton University {awettig,tianyug,zzhong,danqic}@cs.princeton.edu Abstract Masked language models (MLMs) convention- ally mask 15% of tokens due to the belief that more masking would leave insufficient con- text to learn good representations; this mask- ing rate has been widely used, regardless of model sizes or masking strategies. In this work, we revisit this important choice of MLM pre-training. We first establish that 15% is not universally optimal, and larger models should adopt a higher masking rate. Specifi- cally, we find that masking 40% outperforms 15% for BERT-large size models on GLUE and SQuAD. Interestingly, an extremely high masking rate of 80% can still preserve 95% fine-tuning performance and most of the ac- curacy in linguistic probing, challenging the conventional wisdom about the role of the masking rate. We then examine the interplay between masking rates and masking strate- gies and find that uniform masking requires a higher masking rate compared to sophisticated masking strategies such as span or PMI mask- ing. Finally, we argue that increasing the mask- ing rate has two distinct effects: it leads to more corruption, which makes the prediction task harder; it also enables more predictions, which benefits optimization. Using this frame- work, we revisit BERT’s 80-10-10 corruption strategy. Together, our results contribute to a better understanding of MLM pre-training.1 1 Introduction Pre-trained language models have transformed the landscape of natural language processing (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020, inter alia). They are trained on vast quantities of text data and acquire rich and versatile language representations. Compared to autoregressive models, which always predict the next token in a sequence, masked language models *The first two authors contributed equally. 1Our code and pre-trained models are publicly available at https://github.com/princeton-nlp/DinkyTrain. (MLMs) like BERT (Devlin et al., 2019) predict a masked subset of input tokens based on the remain- ing context and are more effective on downstream tasks due to their bidirectional nature. BERT chooses a 15% masking rate, based on the reasoning that models cannot learn good rep- resentations when too much text is masked, and the training is inefficient when too little is masked. Surprisingly, this important choice has been under- explored since 15% masking is used ubiquitously by BERT’s successors (Liu et al., 2019; Joshi et al., 2020; Lan et al., 2020; He et al., 2021; Levine et al., 2021; Izsak et al., 2021), regardless of model sizes, masking strategies and optimization recipes.2 In this work, we aim to understand the impact of masking rates. We hypothesize that the optimal masking rate is not universally 15%, but should depend on other factors. First, we consider the impact of model sizes and establish that indeed larger models should adopt higher masking rates (§3). …

Content truncated to 500 words