Extension of Zipf’s Law to Word and Character N-grams for English and Chinese

Computational Linguistics and Chinese Language Processing Vol. 8, No. 1, February 2003 , pp. 77-102 77  The Association for Computational Linguistics and Chinese Language Processing Extension of Zipf's Law to Word and Character N-grams for English and Chinese Le Quan Ha*, E. I. Sicilia-Garcia*, Ji Ming* and F. J. Smith* Abstract It is shown that for a large corpus, Zipf 's law for both words in English and characters in Chinese does not hold for all ranks. The frequency falls below the frequency predicted by Zipf's law for English words for rank greater than about 5,000 and for Chinese characters for rank greater than about 1,000. However, when single words or characters are combined together with n-gram words or characters in one list and put in order of frequency, the frequency of tokens in the combined list follows Zipf’s law approximately with the slope close to -1 on a log- log plot for all n-grams, down to the lowest frequencies in both languages. This behaviour is also found for English 2-byte and 3-byte word fragments. It only happens when all n-grams are used, including semantically incomplete n-grams. Previous theories do not predict this behaviour, possibly because conditional probabilities of tokens have not been properly represented. Keywords: Zipf 's law, Chinese character, Chinese compound word, n-grams, phrases. 1. Introduction 1.1 Zipf's law The law discovered empirically by [Zipf 1949] for word tokens in a corpus states that if f is the frequency of a word in the corpus and r is the rank, then: f =kr (1) where k is a constant for the corpus. When log(f) is drawn against log(r) in a graph (which is * Computer Science School, Queen's University Belfast, Belfast BT7 1NN, Northern Ireland, UK. Email: {q.le, e.sicilia, j.ming, fj.smith}@qub.ac.uk 78 Le Quan Ha et al. called a Zipf curve), a straight line is obtained with a slope of –1. An example with a small corpus of 250,000 tokens made up of paragraphs chosen at random from the Brown corpus of American English [Francis and Kucera 1964] is given in Figure 1; in this the tokens do not include punctuation marks and numbers. Typographical errors, if any, will appear in the hapax legomenon. -0.2 -0.2 log rank log frequency 1 10 100 1000 10000 100000 100 100000 10 1 10000 1000 Figure 1 Zipf curve for the unigrams extracted from a 250,000-word tokens corpus. Zipf’s discovery was followed by a large body of literature, reviewed in a series of papers edited by [Guiter and Arapov 1982]. Notable among these are papers by [Mandelbrot 1953, 1954, 1959, 1961], [Miller 1954, 1957, 1958], [Simon 1955, 1960, 1961], [Sichel 1975, 1986], [Carroll 1967, 1969], [Baayen 1991], [Chitashvili 1983, 1989] and [Orlov 1983]. It continues to stimulate interest today [Samuelson 1996]; [Baayen 2001]; [Hatzigeorgiu, Mikros and Carayannis 2001]; [Montermurro 2001]; [Ferrer and Solé 2002] and, for example, it has been recently applied to citations [Silagadze 1997], to biological species-abundance [Sichel 1997] and to DNA sequences [Yonezawa and Motohasi 1999]; [Li …

Content truncated to 500 words