The Entropy Rate Principle as a Predictor of Processing Effort: An Evaluation against Eye-tracking Data

The Entropy Rate Principle as a Predictor of Processing Effort: An Evaluation against Eye-tracking Data Frank Keller School of Informatics University of Edinburgh 2 Buccleuch Place Edinburgh EH8 9LW, UK keller@inf.ed.ac.uk Abstract This paper provides evidence for Genzel and Char- niak’s (2002) entropy rate principle, which predicts that the entropy of a sentence increases with its po- sition in the text. We show that this principle holds for individual sentences (not just for averages), but we also find that the entropy rate effect is partly an artifact of sentence length, which also correlates with sentence position. Secondly, we evaluate a set of predictions that the entropy rate principle makes for human language processing; using a corpus of eye-tracking data, we show that entropy and pro- cessing effort are correlated, and that processing ef- fort is constant throughout a text. 1 Introduction Genzel and Charniak (2002, 2003) introduce the en- tropy rate principle, which states that speakers pro- duce language whose entropy rate is on average constant. The motivation for this comes from in- formation theory: the most efficient way of trans- mitting information through a noisy channel is at a constant rate. If human communication has evolved to be optimal in this sense, then we would expect humans to produce text and speech with approx- imately constant entropy. There is some evidence that this is true for speech (Aylett, 1999). For text, the entropy rate principle predicts that the entropy of an individual sentence increases with its position in the text, if entropy is measured out of context. Genzel and Charniak (2002) show that this prediction is true for the Wall Street Journal corpus, for both function words and for content words. They estimate entropy either using a language model or using a probabilistic parser; the effect can be ob- served in both cases. Genzel and Charniak (2003) extend this results in several ways: they show that the effect holds for different genres (but the effect size varies across genres), and also applies within paragraphs, not only within whole texts. Further- more, they show that the effect can also be ob- tained for language other than English (Russian and Spanish). The entropy rate principle also predicts that a language model that takes context into ac- count should yield lower entropy estimates com- pared to an out of context language model. Genzel and Charniak (2002) show that this prediction holds for caching language models such as the ones pro- posed by Kuhn and de Mori (1990). The aim of the present paper is to shed further light on the entropy rate effect discovered by Gen- zel and Charniak (2002, 2003) (henceforth G&C) by providing new evidence in two areas. In Experiment 1, we replicate G&C’s entropy rate effect and investigate the source of the effect. The results show that the correlation coefficients that G&C report are inflated by averaging over sen- tences with the same position, and by restricting the range of the sentence position considered. Once these restrictions are removed …

Content truncated to 500 words