On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5131–5146 June 6–11, 2021. ©2021 Association for Computational Linguistics 5131 On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies Tianyi Zhang Computer Science Department Stanford University tianyizhang@cs.stanford.edu Tatsunori B. Hashimoto Computer Science Department Stanford University thashim@stanford.edu Abstract We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream perfor- mance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained perfor- mance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the masked language model (MLM) objective and existing methods for learning statistical depen- dencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful induc- tive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the im- plied statistical dependence structure outper- forms a classic method for unsupervised pars- ing (58.74 vs. 55.91 UUAS). 1 Introduction Pretrained masked language models (Devlin et al., 2019; Liu et al., 2019b) have benefitted a wide range of natural language processing (NLP) tasks (Liu, 2019; Wadden et al., 2019; Zhu et al., 2020). Despite recent progress in understanding what useful information is captured by MLMs (Liu et al., 2019a; Hewitt and Manning, 2019), it re- mains a mystery why task-agnostic masking of words can capture linguistic structures and transfer to downstream tasks. One popular justification of MLMs relies on viewing masking as a form of cloze reduction. Cloze-like Masking I this movie [MASK] Dependency Learning I like this movie Figure 1: We study the inductive bias of MLM ob- jectives and show that cloze-like masking (left) does not account for much of the downstream performance gains. Instead, we show that MLM objectives are bi- ased towards extracting both statistical and syntactic dependencies using random masks (right). Cloze reductions reformulate an NLP task into a prompt question and a blank and elicit answers by filling in the blank (Figure 1). When tested by cloze reductions pretrained MLMs and left-to-right lan- guage models (LMs) have been shown to possess abundant factual knowledge (Petroni et al., 2019) and display impressive few-shot ability (Brown et al., 2020). This success has inspired recent hypotheses that some word masks are cloze-like and provide indirect supervision to downstream tasks (Saunshi et al., 2020; Lee et al., 2020). For example, a sentiment classification task (Pang et al., 2002) can be reformulated into filling in like …

Content truncated to 500 words