An Iterative Algorithm to Build Chinese Language Models

An Iterative Algorithm to Build Chinese Language Models Xiaoqiang Luo Center for Language and Speech Processing The Johns Hopkins University 3400 N. Charles St. Baltimore, MD21218, USA xiao@j hu. edu Salim Roukos IBM T. J. Watson Research Center Yorktown Heights, NY 10598, USA roukos©wat son. ibm. com Abstract ° • We present an iterative procedure to build a Chinese language model (LM). We seg- ment Chinese text into words based on a word-based Chinese language model. How- ever, the construction of a Chinese LM it- self requires word boundaries. To get out of the chicken-and-egg problem, we propose an iterative procedure that alternates two operations: segmenting text into words and building an LM. Starting with an initial segmented corpus and an LM based upon it, we use a Viterbi-liek algorithm to seg- ment another set of data. Then, we build an LM based on the second set and use the resulting LM to segment again the first cor- pus. The alternating procedure provides a self-organized way for the segmenter to de- tect automatically unseen words and cor- rect segmentation errors. Our prelimi- nary experiment shows that the alternat- ing procedure not only improves the accu- racy of our segmentation, but discovers un- seen words surprisingly well. The resulting word-based LM has a perplexity of 188 for a general Chinese corpus. 1 Introduction In statistical speech recognition(Bahl et al., 1983), it is necessary to build a language model(LM) for as- signing probabilities to hypothesized sentences. The LM is usually built by collecting statistics of words over a large set of text data. While doing so is straightforward for English, it is not trivial to collect statistics for Chinese words since word boundaries are not marked in written Chinese text. Chinese is a morphosyllabic language (DeFrancis, 1984) in that almost all Chinese characters represent a single syllable and most Chinese characters are also mor- phemes. Since a word can be multi-syllabic, it is gen- erally non-trivial to segment a Chinese sentence into words(Wu and Tseng, 1993). Since segmentation is a fundamental problem in Chinese information pro- cessing, there is a large literature to deal with the problem. Recent work includes (Sproat et al., 1994) and (Wang et al., 1992). In this paper, we adopt a statistical approach to segment Chinese text based on an LM because of its autonomous nature and its capability to handle unseen words. As far as speech recognition is concerned, what is needed is a model to assign a probability to a string of characters. One may argue that we could bypass the segmentation problem by building a character- based LM. However, we have a strong belief that a word-based LM would be better than a character- based 1 one. In addition to speech recognition, the use of word based models would have value in infor- mation retrieval and other language processing ap- plications. If word boundaries are given, all established tech- niques can be exploited to construct an LM (Jelinek et al., 1992) just as is done …

Content truncated to 500 words