An Approach to Improve the Smoothing Process Based on Non-uniform Redistribution

Proceedings of PACLIC 19, the 19th Asia-Pacific Conference on Language, Information and Computation. An Approach to Improve the Smoothing Process Based on Non-uniform Redistribution Feng-Long Huang Department of Computer Science and Information Engineering National United University MaioLi, 360, Taiwan flhuang@nuu.edu.tw Ming-Shing Yu * Department of Computer Science, National Chung-Hsing University, Taichung 402, Taiwan msyu@nchu.edu.tw Abstract In the paper, an effective technique, based on the non-uniform redistribution probability for novel events (the unknown events), to improve the smoothing method in language models is proposed. Basically, there are two processes in the smoothing methods: 1) discounting and 2) redistributing. Instead of uniform probability assignment to each unseen events used by most smoothing methods, we propose new technique to improve the redistribution process. Referring to the probabilistic behavior of all seen events, the redistribution process for novel events in our method is non-uniform. The proposed technique is exploited on well-known and frequently-used Good-Turing smoothing method. The empirical results are demonstrated and analyzed for two n-gram models. The improvement is apparent and effective for smoothing methods, especially on higher unseen event rate. Keywords: Language model, Smoothing method, Good-Turing, Cross entropy, Non-uniform Redistribution. 1. Introduction 1.1 Statistical language Models In many domains of natural language processing (NLP); such as speech recognition [1], grammar parser [4], document retrieval [17] and machine translation [Brown]; the statistical language models (LMs) [6], [10] plays an important role in natural language processing. The LMs can be exploited, for instance, to decide the correct target word sequence w. As shown in Fig. 1 of a speech recognition system, the P(W) is the conditional probability of a word sequence W given a speech data S, where W=w1w2w…wm is a possible translation of texts, m is word number of M. The predicted sequence w can be expressed: ) )( argmax(| |) argmax( PSWPW PWS W w w ∂ = = (1) where ) | ( SW P∂ is the probability of input speech given a word sequence W. A language model is regarded as the probability distribution over events or token sequences (texts) that models how often each sequence occurs as a sentence. Chain rule is used to decompose probability prediction: * Correspondence author. ) | )...( )(| ()(| ...) ( ) ( 11 12 3 11 2 1 12 1 − = = m m m m PwPwwPwwPww Pwww w P ). (| ) ( 2 11 1 ∏ = − = m i Pwi wi w P (2) where m w1 denotes the word sequence with m words. Speech S selected words w α P( ) Fig. 1: LMs in speech recognition system. 1.2 n-gram Model Because of the finite training corpora in real world and to reduce the parameter space of word feature in languages, the approximate probability of a given word by using the (n-1)th preceding words is used to estimate sequence W. The probability model with various n can be written: (3) where wi-n+1 denotes the history of n-1 word for word wi. In many applications, the models …

Content truncated to 500 words