Converting Continuous-Space Language Models into N-Gram Language Models for Statistical Machine Translation

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 845–850, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Converting Continuous-Space Language Models into N-gram Language Models for Statistical Machine Translation Rui Wang1,2,3, Masao Utiyama2, Isao Goto2, Eiichro Sumita2, Hai Zhao1,3 and Bao-Liang Lu1,3 1 Center for Brain-Like Computing and Machine Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong Unviersity, Shanghai, 200240, China 2 Multilingual Translation Laboratory, MASTAR Project, National Institute of Information and Communications Technology 3-5 Hikaridai, Keihanna Science City, Kyoto, 619-0289, Japan 3 MOE-Microsoft Key Lab. for Intelligent Computing and Intelligent Systems Shanghai Jiao Tong Unviersity, Shanghai 200240 China wangrui.nlp@gmail.com, mutiyama/igoto/eiichiro.sumita@nict.go.jp, zhaohai@cs.sjtu.edu.cn, bllu@sjtu.edu.cn Abstract Neural network language models, or continuous-space language models (CSLMs), have been shown to improve the performance of statistical machine translation (SMT) when they are used for reranking n-best translations. However, CSLMs have not been used in the first pass decoding of SMT, because using CSLMs in decoding takes a lot of time. In contrast, we propose a method for converting CSLMs into back-off n-gram language models (BNLMs) so that we can use converted CSLMs in decoding. We show that they outperform the original BNLMs and are comparable with the traditional use of CSLMs in reranking. 1 Introduction Language models are important in natural language processing tasks such as speech recognition and statistical machine translation. Traditionally, back- off n-gram language models (BNLMs) (Chen and Goodman, 1996; Chen and Goodman, 1998; Stolcke, 2002) are being widely used for these tasks. Recently, neural network language models, or continuous-space language models (CSLMs) (Bengio et al., 2003; Schwenk, 2007; Le et al., 2011) are being used in statistical machine translation (SMT) (Schwenk et al., 2006; Son et al., 2010; Schwenk et al., 2012; Son et al., 2012; Niehues and Waibel, 2012). These works have shown that CSLMs can improve the BLEU (Papineni et al., 2002) scores of SMT when compared with BNLMs, on the condition that the training data for language modeling are the same size. However, in practice, CSLMs have not been widely used in SMT. One reason is that the computational costs of training and using CSLMs are very high. Various methods have been proposed to tackle the training cost issues (Son et al., 2010; Schwenk et al., 2012; Mikolov et al., 2011). However, there has been little work on reducing using costs. Since the using costs of CSLMs are very high, it is difficult to use CSLMs in decoding directly. A common approach in SMT using CSLMs is the two pass approach, or n-best reranking. In this approach, the first pass uses a BNLM in decoding to produce an n-best list. Then, a CSLM is used to rerank those n-best translations in the second pass. (Schwenk et al., 2006; Son et al., 2010; Schwenk et al., 2012; Son et al., 2012) Another approach is using restricted Boltzmann machines (RBMs) (Niehues and Waibel, 2012) instead of using multi-layer neural networks (Bengio et al., 2003; Schwenk, 2007; Le et al., …

Content truncated to 500 words