Entropy Rate Constancy in Text

Entropy Rate Constancy in Text Dmitriy Genzel and Eugene Charniak Brown Laboratory for Linguistic Information Processing Department of Computer Science Brown University Providence, RI, USA, 02912 {dg,ec}@cs.brown.edu Abstract We present a constancy rate princi- ple governing language generation. We show that this principle implies that lo- cal measures of entropy (ignoring con- text) should increase with the sentence number. We demonstrate that this is indeed the case by measuring entropy in three different ways. We also show that this effect has both lexical (which words are used) and non-lexical (how the words are used) causes. 1 Introduction It is well-known from Information Theory that the most efficient way to send information through noisy channels is at a constant rate. If humans try to communicate in the most efficient way, then they must obey this principle. The communication medium we examine in this pa- per is text, and we present some evidence that this principle holds here. Entropy is a measure of information first pro- posed by Shannon (1948). Informally, entropy of a random variable is proportional to the diffi- culty of correctly guessing the value of this vari- able (when the distribution is known). Entropy is the highest when all values are equally prob- able, and is lowest (equal to 0) when one of the choices has probability of 1, i.e. deterministi- cally known in advance. In this paper we are concerned with entropy of English as exhibited through written text, though these results can easily be extended to speech as well. The random variable we deal with is therefore a unit of text (a word, for our purposes1) that a random person who has pro- duced all the previous words in the text stream is likely to produce next. We have as many ran- dom variables as we have words in a text. The distributions of these variables are obviously dif- ferent and depend on all previous words pro- duced. We claim, however, that the entropy of these random variables is on average the same2. 2 Related Work There has been work in the speech community inspired by this constancy rate principle. In speech, distortion of the audio signal is an extra source of uncertainty, and this principle can by applied in the following way: A given word in one speech context might be common, while in another context it might be rare. To keep the entropy rate constant over time, it would be necessary to take more time (i.e., pronounce more carefully) in less common situations. Aylett (1999) shows that this is in- deed the case. It has also been suggested that the principle of constant entropy rate agrees with biological evidence of how human language processing has evolved (Plotkin and Nowak, 2000). Kontoyiannis (1996) also reports results on 5 consecutive blocks of characters from the works 1It may seem like an arbitrary choice, but a word is a natural unit of length, after all when one is asked to give the length of an essay one typically …

Content truncated to 500 words