Zipf’s Law and Statistical Data on Modern Tibetan

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 322–333, Dublin, Ireland, August 23-29 2014. Zipf’s Law and Statistical Data on Modern Tibetan Huidan Liu Institute of Software, Chinese Academy of Sciences, Beijing, China, 100190 huidan@iscas.ac.cn Minghua Nuo Institute of Software, Chinese Academy of Sciences, Beijing, China, 100190 minghua@iscas.ac.cn Jian Wu Institute of Software, Chinese Academy of Sciences, Beijing, China, 100190 wujian@iscas.ac.cn Abstract In this paper, a large scale modern Tibetan text corpus is built, which includes about 190 thou- sands documents, 67.21 million words, 93.66 million syllables in total. Based on the corpus, statistics are made in several language units in different granularities. Statistical data show that : a syllable has 3.26 letters or 2.20 super characters in average, while a sentence has 75.40 let- ters or 63.14 super characters. The top 10 super characters, syllables, words take up 66.3156%, 16.5556%, 24.6415% of the corpus respectively. Curves for the n-gram frequency-rank list of super chars, syllables and words are plotted. It shows that when all the n-gram phrases for n = 1, 2, . . . , 5 are put together and sorted by frequency in descending order, the frequency-rank curves in log-log axes can be fitted well by a straight line for the unit of syllable and word respec- tively. But for the unit of super character, we didn’t find a curve that can be fitted well enough by a straight line even if we combine all the n-grams for n = 1, 2, . . . , 10. 1 Introduction The statistical property is the natural property of a language. In recent tens of years, people made sta- tistical analysis on Tibetan characters or syllables. But it’s difficult to make statistics on larger language units such as word and n-gram word phrases, especially on a large scale corpus. There are two reasons resulting in the difficulty. First, as Tibetan is a resource poor language, it’s hard to build a large scale Tibetan text corpus. Second, Tibetan word segmentation technology is not well developed even until now. In this paper, we report our word on the statistics on Tibetan based on the language units such as character, syllable, word and their n-gram pairs on a large scale corpus. The remainder of the paper is organized as follow. Tibetan language units are introduced in Section 2. We recall the related work in Section 3. In Section 4, the methods which is used to build the corpus and to segment Tibetan text into language units are described in detail. We make statistics on the corpus and list the most frequency Tibetan language units and test Zipf’s law respectively in Section 5. Section 6 concludes this paper. 2 Language Units of Tibetan Generally speaking, Tibetan is a alphabetic writing system. But there is a unit larger than letter but smaller than syllable, which is different from other language such as English and Chinese. Meanwhile, people have used different terms (in English) to express the same unit …

Content truncated to 500 words