Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888–2913 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2888 Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little Koustuv Sinha†‡ Robin Jia† Dieuwke Hupkes† Joelle Pineau†‡ Adina Williams† Douwe Kiela† † Facebook AI Research; ‡ McGill University / Mila - Quebec AI {koustuvs,adinawilliams,dkiela}@fb.com Abstract A possible explanation for the impressive per- formance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks mostly due to their ability to model higher-order word co- occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and we show that these models still achieve high accuracy after fine- tuning on many downstream tasks - including tasks specifically designed to be challenging for models that ignore word order. Our mod- els also perform surprisingly well according to some parametric syntactic probes, indicat- ing possible deficiencies in how we test repre- sentations for syntactic information. Overall, our results show that purely distributional in- formation largely explains the success of pre- training, and they underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge. 1 Introduction The field of natural language processing (NLP) has become dominated by the pretrain-and-finetune paradigm, where we first obtain a good paramet- ric prior in order to subsequently model down- stream tasks accurately. In particular, masked lan- guage model (MLM) pre-training, as epitomized by BERT (Devlin et al., 2019), has proven wildly suc- cessful, although the precise reason for this success has remained unclear. On one hand, we can view BERT as the newest in a long line of NLP tech- niques (Deerwester et al., 1990; Landauer and Du- mais, 1997; Collobert and Weston, 2008; Mikolov et al., 2013; Peters et al., 2018) that exploit the well- known distributional hypothesis (Harris, 1954).1 On the other hand, it has been claimed that BERT 1One might even argue that BERT is not actually all that different from earlier distributional models like word2vec (Mikolov et al., 2013), see Appendix A. “rediscovers the classical NLP pipeline” (Tenney et al., 2019), suggesting that it has learned “the types of syntactic and semantic abstractions tradi- tionally believed necessary for language process- ing” rather than “simply modeling complex co- occurrence statistics” (ibid. p.1). In this work, we aim to uncover how much of MLM’s success comes from learning simple distri- butional information, as opposed to grammatical abstractions (Tenney et al., 2019; Manning et al., 2020). We disentangle these two hypotheses by measuring the effect of removing word order in- formation during pre-training: any sophisticated (English) NLP pipeline would presumably depend on the syntactic information conveyed by the or- der of words. Surprisingly, we find that most of MLM’s high performance can in fact be explained by the “distributional prior” rather than its ability to replicate the classical NLP …

Content truncated to 500 words