Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 609–619 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 609 Cloze Distillation: Improving Neural Language Models with Human Next-Word Predictions Tiwalayo N. Eisape1 Noga Zaslavsky1,2 1Department of Brain and Cognitive Sciences, 2Center for Brains Minds and Machines Massachusetts Institute of Technology {eisape, nogazs, rplevy}@mit.edu Roger P. Levy1 Abstract Contemporary autoregressive language mod- els (LMs) trained purely on corpus data have been shown to capture numerous features of human incremental processing. However, past work has also suggested dissociations between corpus probabilities and human next-word pre- dictions. Here we evaluate several state-of-the- art language models for their match to human next-word predictions and to reading time be- havior from eye movements. We then propose a novel method for distilling the linguistic in- formation implicit in human linguistic predic- tions into pre-trained LMs: Cloze Distillation. We apply this method to a baseline neural LM and show potential improvement in reading time prediction and generalization to held-out human cloze data. 1 Introduction Modern language models (LMs) demonstrate out- standing general-purpose command over language. The majority of these models acquire language by maximizing the in-context probability of each word in their training corpus (Figure 1), typically with a self-supervised objective. This simple corpus probability matching has resulted in models that learn impressive powers of both psychometric pre- diction (Frank and Bod, 2011; Fossum and Levy, 2012; Frank et al., 2015; Goodkind and Bicknell, 2018; Hale et al., 2018; van Schijndel and Linzen, 2018; Warstadt and Bowman, 2020; Wilcox et al., 2020) and language more generally (Devlin et al., 2019; Radford et al., 2019). In humans, prediction may underlie both learn- ing (Kuhl, 2004; Huang and Snedeker, 2013) and processing (Ryskin et al., 2020; Levy, 2008; Clark, 2013). Human linguistic prediction can be under- stood as not only lexical but also as taking place both above and below the word level (Federmeier and Kutas, 1999; Federmeier et al., 2002); parallel, i.e., predictive commitments are maintained over several linguistic units at once (Levy, 2008); and graded, i.e., commitment is licensed to varying de- grees based on features of the linguistic unit being predicted. Rather than placing bets (Jackendoff, 1987) on which single word will come next, hu- mans make many diffuse bets at multiple linguistic levels (e.g., syntactic, orthographic, lexical, etc.). Surprisal theory (Hale, 2001; Levy, 2008) de- scribes the utility of the approach taken by the human language processor, as lexical prediction is often an ill-constrained classification problem — for agents with very large vocabularies (LMs, humans), context is often not sufficiently constrain- ing for high accuracy multiple, thousand-way clas- sification decisions, but is typically constraining enough to accurately infer next-word features (such as part of speech, and semantic category). A large body of evidence demonstrates that these graded next-word predictions are reflected in human pro- cessing times (Ehrlich and Rayner, 1981; Demberg and Keller, 2008; Smith and Levy, 2013; Luke and Christianson, 2016) as well as neural responses (Kutas and Hillyard, 1980; …

Content truncated to 500 words