On the Influence of Masking Policies in Intermediate Pre-training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7190–7202 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7190 On the Influence of Masking Policies in Intermediate Pre-training Qinyuan Ye1† Belinda Z. Li2‡ Sinong Wang3 Benjamin Bolte3 Hao Ma3 Wen-tau Yih3 Xiang Ren1 Madian Khabsa3 1University of Southern California 2MIT CSAIL 3Facebook AI {qinyuany,xiangren}@usc.edu bzl@mit.edu {sinongwang,bbolte,scottyih,haom,mkhabsa}@facebook.com Abstract Current NLP models are predominantly trained through a two-stage “pre-train then fine-tune” pipeline. Prior work has shown that inserting an intermediate pre-training stage, using heuristic masking policies for masked language modeling (MLM), can significantly improve final performance. However, it is still unclear (1) in what cases such inter- mediate pre-training is helpful, (2) whether hand-crafted heuristic objectives are optimal for a given task, and (3) whether a masking policy designed for one task is generalizable beyond that task. In this paper, we perform a large-scale empirical study to investigate the effect of various masking policies in intermediate pre-training with nine selected tasks across three categories. Crucially, we introduce methods to automate the discov- ery of optimal masking policies via direct supervision or meta-learning. We conclude that the success of intermediate pre-training is dependent on appropriate pre-train corpus, selection of output format (i.e., masked spans or full sentence), and clear understanding of the role that MLM plays for the downstream task. In addition, we find our learned masking policies outperform the heuristic of masking named entities on TriviaQA, and policies learned from one task can positively transfer to other tasks in certain cases, inviting future research in this direction. 1 Introduction Large, neural language models (LMs) pre-trained with masked language modeling (Devlin et al., 2019; Raffel et al., 2020) have achieved impressive results over a variety of NLP tasks. Studies show that an additional intermediate pre-training stage between general pre-training and task-specific fine- tuning further improves downstream performance (Fig. 1). For example, intermediate pre-training †Work partially done while interning at Facebook AI. ‡Work partially done while working at Facebook AI. Source: In Newtonian physics, free fall <mask> motion of a <mask> where gravity is <mask> only <mask> upon it. Target: In Newtonian physics, free fall is any motion of a body where gravity is the only force acting upon it. Target: gravity Source: which force acts on an object in free fall General Pre-training with Random Masks Intermediate Pre-training To do or not to do? with Heuristic or Learned Masks? Our Focus Task-specific Fine-tuning Pre- trained Model Pre- trained Model Source: In Newtonian physics, free fall is any motion of a body where <mask> is the only force acting upon it. Pre- trained Model Target: gravity Figure 1: Analysis Setup. We investigate the influence brought by different masking policies during interme- diate pre-training, a stage between general pre-training and task-specific fine-tuning. We apply three types of policies (heuristic, supervised, meta-learned) on three categories of tasks (closed-book QA, knowledge- intensive language tasks, multiple-choice QA). by masking and recovering named entities or dates, known as salient span masking (SSM, Guu et al. …

Content truncated to 500 words