Reduced N-Grams for Chinese Evaluation

Computational Linguistics and Chinese Language Processing Vol. 10, No. 1, March 2005, pp. 19-34 19 © The Association for Computational Linguistics and Chinese Language Processing Reduced N-Grams for Chinese Evaluation Le Quan Ha*, R. Seymour+, P. Hanna*and F. J. Smith* Abstract Theoretically, an improvement in a language model occurs as the size of the n-grams increases from 3 to 5 or higher. As the n-gram size increases, the number of parameters and calculations, and the storage requirement increase very rapidly if we attempt to store all possible combinations of n-grams. To avoid these problems, the reduced n-grams’ approach previously developed by O’ Boyle and Smith [1993] can be applied. A reduced n-gram language model, called a reduced model, can efficiently store an entire corpus’s phrase-history length within feasible storage limits. Another advantage of reduced n-grams is that they usually are semantically complete. In our experiments, the reduced n-gram creation method or the O’ Boyle-Smith reduced n-gram algorithm was applied to a large Chinese corpus. The Chinese reduced n-gram Zipf curves are presented here and compared with previously obtained conventional Chinese n-grams. The Chinese reduced model reduced perplexity by 8.74% and the language model size by a factor of 11.49. This paper is the first attempt to model Chinese reduced n-grams, and may provide important insights for Chinese linguistic research. Keywords: Reduced n-grams, reduced n-gram algorithm / identification, reduced model, Chinese reduced n-grams, Chinese reduced model 1. Introduction to the Reduced N-Gram Approach P O’ Boyle and F J Smith [1992, 1993] proposed a statistical method to improve language models based on the removal of overlapping phrases. The distortion of phrase frequencies were first observed in the Vodis Corpus when the bigram “RAIL ENQUIRIES” and its super-phrase “BRITISH RAIL ENQUIRIES” were examined and reported by O’ Boyle. Both occur 73 times, which is a large number for such a small corpus. “ENQUIRIES” follows “RAIL” with a very high probability when it is preceded by “BRITISH.” However, when “RAIL” is preceded by words other than “BRITISH,” “ENQUIRIES” does not occur, but words like “TICKET” or “JOURNEY” may. Thus, the * Computer Science School, Queen's University Belfast, Belfast BT7 1NN, Northern Ireland, UK. Email: {q.le, p.hanna, fj.smith}@qub.ac.uk + Email: rowan@rowan.ws 20 Le Quan Ha et al. bigram “RAIL ENQUIRIES” gives a misleading probability that “RAIL” is followed by “ENQUIRIES” irrespective of what precedes it. At the time of their research, Smith and O’ Boyle reduced the frequencies of “RAIL ENQUIRIES” by using the frequency of the larger trigram, which gave a probability of zero for “ENQUIRIES” following “RAIL” if it was not preceded by “BRITISH.” This problem happens not only with word-token corpora but also corpora in which all the compounds are tagged as a unit since overlapping n-grams still appear. Therefore, a phrase can occur in a corpus as a reduced n-gram in some places and as part of a larger reduced n-gram in other places. In a reduced model, the occurrence of an n-gram is not counted when it is a …

Content truncated to 500 words