Variation of Entropy and Parse Trees of Sentences as a Function of the Sentence Number

Variation of Entropy and Parse Trees of Sentences as a Function of the Sentence Number Dmitriy Genzel and Eugene Charniak Brown Laboratory for Linguistic Information Processing Department of Computer Science Brown University Providence, RI, USA, 02912 {dg,ec}@cs.brown.edu Abstract In this paper we explore the variation of sentences as a function of the sentence number. We demonstrate that while the entropy of the sentence increases with the sentence number, it decreases at the para- graph boundaries in accordance with the Entropy Rate Constancy principle (intro- duced in related work). We also demon- strate that the principle holds for differ- ent genres and languages and explore the role of genre informativeness. We investi- gate potential causes of entropy variation by looking at the tree depth, the branch- ing factor, the size of constituents, and the occurrence of gapping. 1 Introduction and Related Work In many natural language processing applications, such as parsing or language modeling, sentences are treated as natural self-contained units. Yet it is well- known that for interpreting the sentences the dis- course context is often very important. The later sentences in the discourse contain references to the entities in the preceding sentences, and this fact is often useful, e.g., in caching for language model- ing (Goodman, 2001). The indirect influence of the context, however, can be observed even when a sen- tence is taken as a stand-alone unit, i.e., without its context. It is possible to distinguish between a set of earlier sentences and a set of later sentences with- out any direct comparison by computing certain lo- cal statistics of individual sentences, such as their entropy (Genzel and Charniak, 2002). In this work we provide additional evidence for this hypothesis and investigate other sentence statistics. 1.1 Entropy Rate Constancy Entropy, as a measure of information, is often used in the communication theory. If humans have evolved to communicate in the most efficient way (some evidence for that is provided by Plotkin and Nowak (2000)), then they would communicate in such a way that the entropy rate would be constant, namely, equal to the channel capacity (Shannon, 1948). In our previous work (Genzel and Charniak, 2002) we propose that entropy rate is indeed con- stant in human communications. When read in con- text, each sentence would appear to contain roughly the same amount of information, per word, whether it is the first sentence or the tenth one. Thus the tenth sentence, when taken out of context, must ap- pear significantly more informative (and therefore harder to process), since it implicitly assumes that the reader already knows all the information in the preceding nine sentences. Indeed, the greater the sentence number, the harder to process the sentence must appear, though for large sentence numbers this would be very difficult to detect. This makes intu- itive sense: out-of-context sentences are harder to understand than in-context ones, and first sentences can never be out of context. It is also demonstrated empirically through estimating entropy rate of vari- ous sentences. In the first …

Content truncated to 500 words