Neural Network Based Bilingual Language Model Growing for Statistical Machine Translation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 189–195, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Neural Network Based Bilingual Language Model Growing for Statistical Machine Translation Rui Wang1,3,∗, Hai Zhao1,3, Bao-Liang Lu1,3, Masao Utiyama2 and Eiichro Sumita2 1Center for Brain-Like Computing and Machine Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China 2Multilingual Translation Laboratory, MASTAR Project, National Institute of Information and Communications Technology, 3-5 Hikaridai, Keihanna Science City, Kyoto, 619-0289, Japan 3Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China wangrui.nlp@gmail.com, {zhaohai, blu}@cs.sjtu.edu.cn, {mutiyama, eiichiro.sumita}@nict.go.jp Abstract Since larger n-gram Language Model (LM) usually performs better in Statistical Machine Translation (SMT), how to con- struct efficient large LM is an important topic in SMT. However, most of the ex- isting LM growing methods need an extra monolingual corpus, where additional LM adaption technology is necessary. In this paper, we propose a novel neural network based bilingual LM growing method, only using the bilingual parallel corpus in SMT. The results show that our method can im- prove both the perplexity score for LM e- valuation and BLEU score for SMT, and significantly outperforms the existing LM growing methods without extra corpus. 1 Introduction ‘Language Model (LM) Growing’ refers to adding n-grams outside the corpus together with their probabilities into the original LM. This operation is useful as it can make LM perform better through letting it become larger and larger, by only using a small training corpus. There are various methods for adding n-grams selected by different criteria from a monolingual corpus (Ristad and Thomas, 1995; Niesler and Woodland, 1996; Siu and Ostendorf, 2000; Si- ivola et al., 2007). However, all of these approach- es need additional corpora. Meanwhile the extra corpora from different domains will not result in better LMs (Clarkson and Robinson, 1997; Iyer et al., 1997; Bellegarda, 2004; Koehn and Schroeder, ∗Part of this work was done as Rui Wang visited in NICT. 2007). In addition, it is very difficult or even im- possible to collect an extra large corpus for some special domains such as the TED corpus (Cettolo et al., 2012) or for some rare languages. There- fore, to improve the performance of LMs, without assistance of extra corpus, is one of important re- search topics in SMT. Recently, Continues Space Language Model (CSLM), especially Neural Network based Lan- guage Model (NNLM) (Bengio et al., 2003; Schwenk, 2007; Mikolov et al., 2010; Le et al., 2011), is being actively used in SMT (Schwenk et al., 2006; Son et al., 2010; Schwenk, 2010; Schwenk et al., 2012; Son et al., 2012; Niehues and Waibel, 2012). One of the main advantages of CSLM is that it can more accurately predic- t the probabilities of the n-grams, which are not in the training corpus. However, in practice, CSLM- s have not been widely used in the current SMT systems, due to their too high …

Content truncated to 500 words