A Maximum Entropy Approach for Semantic Language Modeling

Computational Linguistics and Chinese Language Processing Vol. 11, No. 1, March 2006, pp. 37-56 37 © The Association for Computational Linguistics and Chinese Language Processing [Received June 16, 2005; Revised January 2, 2006; Accepted January 3, 2006] A Maximum Entropy Approach for Semantic Language Modeling Chuang-Hua Chueh∗, Hsin-Min Wang+ and Jen-Tzung Chien∗ Abstract The conventional n-gram language model exploits only the immediate context of historical words without exploring long-distance semantic information. In this paper, we present a new information source extracted from latent semantic analysis (LSA) and adopt the maximum entropy (ME) principle to integrate it into an n-gram language model. With the ME approach, each information source serves as a set of constraints, which should be satisfied to estimate a hybrid statistical language model with maximum randomness. For comparative study, we also carry out knowledge integration via linear interpolation (LI). In the experiments on the TDT2 Chinese corpus, we find that the ME language model that combines the features of trigram and semantic information achieves a 17.9% perplexity reduction compared to the conventional trigram language model, and it outperforms the LI language model. Furthermore, in evaluation on a Mandarin speech recognition task, the ME and LI language models reduce the character error rate by 16.9% and 8.5%, respectively, over the bigram language model. Keywords: Language Modeling, Latent Semantic Analysis, Maximum Entropy, Speech Recognition 1. Introduction Language modeling plays an important role in automatic speech recognition (ASR). Given a speech signal O , the most likely word sequence ˆW is obtained by maximizing a posteriori probability ) ( pWO , or, equivalently, the product of acoustic likelihood ) ( pOW and prior probability of word sequence ( ) p W : ˆ arg max ( ) arg max ( ) ( ) W W W p W O p O W p W = = . (1) ∗ Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan, Taiwan, R. O. C E-mail: chchueh@chien.csie.ncku.edu.tw + Institute of Information Science, Academia Sinica, Taipei, Taiwan, R. O. C 38 Chuang-Hua Chueh et al. This prior probability corresponds to the language model that is useful in characterizing regularities in natural language. Also, this language model has been widely employed in optical character recognition, machine translation, document classification, information retrieval [Ponte and Croft 1998], and many other applications. In the literature, there were several approaches have been taken to extract different linguistic regularities in natural language. The structural language model [Chelba and Jelinek 2000] extracted the relevant syntactic regularities based on predefined grammar rules. Also, the large-span language model [Bellegarda 2000] was feasible for exploring the document-level semantic regularities. Nevertheless, the conventional n-gram model was effective at capturing local lexical regularities. In this paper, we focus on developing a novel latent semantic n-gram language model for continuous Mandarin speech recognition. When considering an n-gram model, the probability of a word sequence W is written as a product of probabilities of individual words conditioned on their preceding n-1 words 1 1 2 1 …

Content truncated to 500 words