An Effective Combination of Different Order N-grams

An Effective Combination of Different Order N-grams Sen Zhang, Na Dong Speech Group, INRIA-LORIA B.P.101 54602 Villers les Nancy, France zhangsen@yahoo.com Abstract In this paper an approach is proposed to combine different order N-grams based on the discriminative estimation criterion, on which the parameters of n-gram can be optimized. To raise the power of modeling language information, we propose several schemes to combine conventional different order n-gram language model. We employ Newton Gradient method to estimate the assumption probabilities and then test the optimally selected language model. We conduct experiments on the platform of conversion from Chinese pinyin to Chinese character. The experimental results show that the memory capacity of language model can be remarkably lowered with hide loss of accuracy. 1. Introduction In Chinese natural language processing domain, the parameters of n-gram can be estimated by calculating the frequency of word pair in text corpus and then normalizing the frequency. This conventional language model cannot satisfy our requirements since it is dependent of discriminative capability. We propose discriminative estimation approach, which can directly relate the estimation of n-gram parameters to its discriminative capability. We optimize the parameters of n-gram on the criterion of discriminative estimation by using Newton Gradient method. When we establish N-gram we artificially introduce an assumption over the relationship among adjacent words. Uni-gram is based upon the assumption that all words appear in the corpus independently. Bi-gram assumes that only contiguous words correlate with each other and tri-gram puts a constraint on the language information that one word can be predicted only by its two predecessor words. Some words are free of context and some depend on short or long history information under some circumstances. In this sense single N-gram could model the language phenomena with some compromise. This paper addresses the impact of the different assumption from different order n-gram on the performance of language model and proposed the combination and optimal selection of different n-gram to battle with artificial assumption and possible data sparsity problem. In the following sections, we first bring up with the discriminative estimation criterion. Next we describe the assumption from N-gram. In the following section we introduce the scheme for combination of different order N-gram. In the next section, an approach to optimal selection of different order language model is proposed. At last we report the experimental results on the platform of conversion from Chinese pinyin to Chinese character. 251 2 Discriminative Estimation In natural language processing, statistical language model has been proved to be a successful method and great efforts have been taken in building n-gram language model. (R. Isotani, 1994) reports the result of the research on stochastic language model with local and global language information. N-gram can be looked as Markov chain model, so maximum likelihood estimation can be used in n- gram similar to the estimation of HMM parameters (L. Bahl, 1983). At the same time, the estimation of n-gram parameters is not surely relative to discriminative capability. Discriminative capability means the power that n-gram gets …

Content truncated to 500 words