Applications and Explanations of Zipf’s Law

II l / I / / / / / / / / Applications and Explanations of Zipf's Law David M. W. Powers Department of Computer Science The Flinders University of South Australia powers@acm.org Abstract Recently I have been intrigued by the reappearance of an old friend, George Kingsley Zipf, in a number of not entirely expected places. The law named for him is ubiquitous, but Zipf did not actually discover the law so much as provide a plausible explanation. Others have proposed modifications to Zipf's Law, and closer examination uncovers systematic deviations from its normative form. We demonstrate how Zipf's analysis can be extended to include some of these phenomena. 1. Introduction and Motivation In this paper we wish to revisit Zipf's study of the relationship between rank and frequency of various linguistic and social units and constructions. The paper arises out of observations in Natural Language Learning experiments of deviations from the received version of Zipf's Law. As it may not be immediately obvious why this relationship is of significance in NLL, we very briefly mention some of the places where the relationship affects research in our field, and which we feel could usefully be further explored. 1.1 Quantitative Linguistics The field of which Zipf was the pioneer is discovering lots of interesting empirical laws, but how much has it advanced in explanation or application (K6hler,1991)? 1.2 Statistical Learning Methods Zipf's Law tells us how much text we have to look at and how precise our statistics have to be to achieve what level of expected error. (Finch, 1993; Powers, 1996). For example, the most frequent 150 words typically account for around half the words of a corpus, although this figure varies significantly with the size of the corpus, the size of the lexicon, the genre, register and medium of communication and the linguistic complexity of the text -- and this is one of the phenomena we wish to start to examine in this paper. Zipf's Law is also closely related to the Good-Turing smoothing technique, and a better law could lead to better smoothing (Samuelsson, 1996). Note that Samuelsson showed that Zipf's Law implies a smoothing function slightly different from Geod-Turing. 1.3 Semantics and Information Retrieval Zipf's Law provides a base-line model for expected occurence of target terms and the answers to certain questions may provide considerable information about its role in the corpus (Steele,1998): What does it mean to ask if a word is significant in a corpus, beyond mere oecurence or relative probability? What is the range of the semantic influence of a word in a corpus? What does the pattern ofoceurences contribute to our assessment of its relevance in the corpus? 1.4 Parser Evaluation Zipf's Law provides a basis for evaluating parsers and taggers (Entwisle and Powers, 1998). Again we summarize the potential role in the form of a series of questions: How does a language model developed on one corpus transfer to another? How do we translate performance estimates on a few test corpora …

Content truncated to 500 words