A Cognitive Regularizer for Language Modeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5191–5202 August 1–6, 2021. ©2021 Association for Computational Linguistics 5191 A Cognitive Regularizer for Language Modeling Jason Wei Clara Meister Ryan Cotterell , Google AI Language ETH Zürich University of Cambridge jasonwei@google.com clara.meister@inf.ethz.ch ryan.cotterell@inf.ethz.ch Abstract The uniform information density (UID) hy- pothesis, which posits that speakers behaving optimally tend to distribute information uni- formly across a linguistic signal, has gained traction in psycholinguistics as an explanation for certain syntactic, morphological, and prosodic choices. In this work, we explore whether the UID hypothesis can be opera- tionalized as an inductive bias for statistical language modeling. Specifically, we augment the canonical MLE objective for training lan- guage models with a regularizer that encodes UID. In experiments on ten languages span- ning five language families, we find that using UID regularization consistently improves perplexity in language models, having a larger effect when training data is limited. Moreover, via an analysis of generated sequences, we find that UID-regularized language models have other desirable properties, e.g., they gen- erate text that is more lexically diverse. Our results not only suggest that UID is a reason- able inductive bias for language modeling, but also provide an alternative validation of the UID hypothesis using modern-day NLP tools. 1 Introduction Language has been hypothesized to follow certain information-theoretic constraints. One of the most famous of these constraints is the uniform infor- mation density (UID) hypothesis (Fenk and Fenk, 1980; Jaeger, 2010), which states that, subject to the rules of the grammar, speakers aim to distribute information density across a linguistic signal as uniformly as possible. That is, speakers behav- ing optimally should structure their utterances such that the differences between the peaks and troughs in information are minimized. In the psycholinguistics literature, the UID hy- pothesis has been used to explain a variety of lin- guistic phenomena ranging from how we shorten the phonetic duration of more-predictable linguistic (a) (b) Figure 1: Graphical illustration of two examples regard- ing UID. In (a), many speakers will prefer the version with the relativizer that (dotted blue line). The UID hypothesis posits that this is because, without the rela- tivizer, the first word of the relative clause, we, has high information density; and so including the relativizer distributes the per-word information density more uni- formly. In (b), the relativizer that is often omitted be- cause, at the onset of the relative clause, the informa- tion density of I is lower and therefore the distribution of information density is already relatively uniform. Il- lustration based on Jaeger (2010). units (Aylett and Turk, 2004) to when we decide to use optional syntactic relativizers (Levy and Jaeger, 2007), among other phenomena (Bell et al., 2003; Frank and Jaeger, 2008). These studies often use language models to estimate the information den- sity of linguistic units, taking observations of low variation of information density in well-formed ut- terances as evidence for the UID …

Content truncated to 500 words