Learning Better Masking for Better Language Model Pre-training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7255–7267 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning Better Masking for Better Language Model Pre-training Dongjie Yang1,2, Zhuosheng Zhang1,2,∗, Hai Zhao1,2,∗ 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University {djyang.tony,zhangzs}@sjtu.edu.cn,zhaohai@cs.sjtu.edu.cn Abstract Masked Language Modeling (MLM) has been widely used as the denoising objective in pre- training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training. However, the model may receive a complicated impact from pre-training status, which changes accordingly as training time goes on. In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings. We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks. Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pre- training1. 1 Introduction Pre-trained language models (PrLMs) have played an essential role in many natural language pro- cessing tasks (Radford et al., 2018; Devlin et al., 2019; Bao et al., 2020; Guu et al., 2020; Yu et al., 2021; Zhang et al., 2022). Generally speaking, PrLMs can be seen as an automatic denoising encoder and may be conveniently obtained through a self-supervised learning way. Masked Language Modeling (MLM) pioneered by BERT (Devlin et al., 2019) is a widely used denoising method for language model pre-training (Lan et al., 2020; Clark et al., 2020). In MLM pre-training, a subset ∗Corresponding author; This paper was partially supported by Key Projects of National Natural Science Foundation of China (U1836222 and 61733011). 1https://github.com/mutonix/better_masking of tokens in a sequence is masked with a certain masking ratio, and the masked sequence is fed to the PrLM, which is required to predict the masked tokens. Masking in MLM is a process in terms of sampling masked tokens from a huge data space to generate training batches, in which MLM may be heavily controlled by two main factors, masking ratio and masked contents. So far, only a few stud- ies have ever considered optimal settings for better MLM from quite limited perspectives. Especially, all known works only take time-invariant MLM setting into account despite the huge time variance of the model during a lengthy pre-training. For example, carefully considered masked units like n- gram, entity and span (Sun et al., 2019; Joshi et al., 2020; Levine et al., 2021; Li and Zhao, 2021) are adopted throughout the entire pre-training. Another example is that exploring a good enough (but still fixed) …

Content truncated to 500 words