A Large Scale Distributed Syntactic, Semantic and Lexical Language Model for Machine Translation

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 201–210, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics A Large Scale Distributed Syntactic, Semantic and Lexical Language Model for Machine Translation Ming Tan Wenli Zhou Lei Zheng Shaojun Wang Kno.e.sis Center Department of Computer Science and Engineering Wright State University Dayton, OH 45435, USA {tan.6,zhou.23,lei.zheng,shaojun.wang}@wright.edu Abstract This paper presents an attempt at building a large scale distributed composite language model that simultaneously accounts for local word lexical information, mid-range sentence syntactic structure, and long-span document semantic content under a directed Markov ran- dom field paradigm. The composite language model has been trained by performing a con- vergent N-best list approximate EM algorithm that has linear time complexity and a follow- up EM algorithm to improve word prediction power on corpora with up to a billion tokens and stored on a supercomputer. The large scale distributed composite language model gives drastic perplexity reduction over n- grams and achieves significantly better trans- lation quality measured by the BLEU score and “readability” when applied to the task of re-ranking the N-best list from a state-of-the- art parsing-based machine translation system. 1 Introduction The Markov chain (n-gram) source models, which predict each word on the basis of previous n-1 words, have been the workhorses of state-of-the-art speech recognizers and machine translators that help to resolve acoustic or foreign language ambiguities by placing higher probability on more likely original underlying word strings. Research groups (Brants et al., 2007; Zhang, 2008) have shown that using an immense distributed computing paradigm, up to 6- grams can be trained on up to billions and trillions of words, yielding consistent system improvements, but Zhang (2008) did not observe much improve- ment beyond 6-grams. Although the Markov chains are efficient at encoding local word interactions, the n-gram model clearly ignores the rich syntactic and semantic structures that constrain natural languages. As the machine translation (MT) working groups stated on page 3 of their final report (Lavie et al., 2006), “These approaches have resulted in small im- provements in MT quality, but have not fundamen- tally solved the problem. There is a dire need for de- veloping novel approaches to language modeling.” Wang et al. (2006) integrated n-gram, structured language model (SLM) (Chelba and Jelinek, 2000) and probabilistic latent semantic analysis (PLSA) (Hofmann, 2001) under the directed MRF frame- work (Wang et al., 2005) and studied the stochas- tic properties for the composite language model. They derived a generalized inside-outside algorithm to train the composite language model from a gen- eral EM (Dempster et al., 1977) by following Je- linek’s ingenious definition of the inside and outside probabilities for SLM (Jelinek, 2004) with 6th order of sentence length time complexity. Unfortunately, there are no experimental results reported. In this paper, we study the same composite lan- guage model. Instead of using the 6th order general- ized inside-outside algorithm proposed in (Wang et al., 2006), we train this composite model by a con- vergent N-best list …

Content truncated to 500 words