Extension of Zipf’s Law to Words and Phrases

Extension of Zipf’s Law to Words and Phrases Le Quan Ha, E. I. Sicilia-Garcia, Ji Ming, F. J. Smith School Computer Science Queen’s University of Belfast Belfast BT7 1NN, Northern Ireland q.le@qub.ac.uk Abstract Zipf’s law states that the frequency of word tokens in a large corpus of natural language is inversely proportional to the rank. The law is investigated for two languages English and Mandarin and for n- gram word phrases as well as for single words. The law for single words is shown to be valid only for high frequency words. However, when single word and n-gram phrases are combined together in one list and put in order of frequency the combined list follows Zipf’s law accurately for all words and phrases, down to the lowest frequencies in both languages. The Zipf curves for the two languages are then almost identical. 1. Introduction The law discovered empirically by Zipf (1949) for word tokens in a corpus states that if f is the frequency of a word in the corpus and r is the rank, then: f =kr (1) where k is a constant for the corpus. When log(f) is drawn against log(r) in a graph (which is often called a Zipf curve), a straight line is obtained with a slope of –1. An example with a small corpus of 250,000 tokens is given in Figure 1. Zipf’s discovery was followed by a large body of literature reviewed in a series of papers edited by Guiter and Arapov (1982). It continues to stimulate interest today (Samuelson, 1996; Montermurro, 2002; Ferrer and Solé, 2002) and, for example, it has been applied to citations Silagadze (1997) and to DNA sequences (Yonezawa & Motohasi, 1999; Li, 2001). 1 10 100 1000 10000 100000 1 10 100 1000 10000 100000 log rank Figure 1 Zipf curve for the unigrams extracted from a 250,000 word tokens corpus Zipf discovered the law by analysing manually the frequencies of words in the novel “Ulysses” by James Joyce. It contains a vocabulary of 29,899 different word types associated with 260,430 word tokens. Following its discovery in 1949, several experiments aided by the appearance of the computer in the 1960’s, confirmed that the law was correct for the small corpora which could be processed at that time. The slope of the curve was found to vary slightly from –1 for some corpora; also the frequencies for the highest ranked words sometimes deviated slightly from the straight line, which suggested several modifications of the law, and in particular one due to Mandelbrot (1953): = r( +α)β k f (2) where α and β are constants for the corpus being analysed. However, generally the constants α and β were found to be only small statistical deviations from the original law by Zipf (exceptions are legal texts which have smaller β values (≈0.9) showing that lawyers use more words than other people!)(Smith & Devine, 1985). A number of theoretical developments of Zipf’s law had been derived in the 50’s and 60’s and …

Content truncated to 500 words