Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963–980 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 963 Revisiting the Uniform Information Density Hypothesis Clara Meister1, Tiago Pimentel2, Patrick Haller3, Lena J¨ager3,4, Ryan Cotterell1,2, Roger Levy5 1ETH Z¨urich 2University of Cambridge 3University of Zurich 4University of Potsdam 5Massachusetts Institute of Technology clara.meister@inf.ethz.ch tp472@cam.ac.uk haller@cl.uzh.ch jaeger@cl.uzh.ch ryan.cotterell@inf.ethz.ch rplevy@mit.edu Abstract The uniform information density (UID) hy- pothesis posits a preference among language users for utterances structured such that infor- mation is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal—or lack thereof—should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID’s predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uni- formity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document—a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.1 1 Introduction The uniform information density (UID) hypothesis (Fenk and Fenk, 1980; Levy and Jaeger, 2007) states that language users prefer when information content (measured information-theoretically as 1Analysis pipeline is publicly available and can be found at https://github.com/rycolab/revisiting-uid. Figure 1: Correlation coefficient between (negative) sum of surprisals raised to the kth power and linguistic acceptability judgments of a sentence. The higher cor- relation when k > 1 implies sentences with a more uni- form distribution of information are more acceptable. surprisal) is distributed as smoothly as possible throughout an utterance. The studies adduced in support of this hypothesis in language production span levels of linguistic structure: from phonetics (Aylett and Turk, 2004) to lexical choice (Ma- howald et al., 2013), to syntax (Jaeger, 2010), and to discourse (Torabi Asr and Demberg 2015) (though see Zhan and Levy 2018, 2019). Despite this evidence, there are several aspects of the UID hypothesis that lack clarity or unity. For example, there is a dearth of converging evidence from studies in language comprehension. Furthermore, multiple candidate operationalizations of UID have been proposed, each without formal justification for their choices (Collins, 2014; Jain et al., 2018; Meister et al., 2020; Wei et al., 2021). In this work, we attempt to shed light on these is- sues: we …

Content truncated to 500 words