Encoding Source Language with Convolutional Neural Network for Machine Translation

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 20–30, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Encoding Source Language with Convolutional Neural Network for Machine Translation Fandong Meng1 Zhengdong Lu2 Mingxuan Wang1 Hang Li2 Wenbin Jiang1 Qun Liu3,1 1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences {mengfandong,wangmingxuan,jiangwenbin,liuqun}@ict.ac.cn 2Noah’s Ark Lab, Huawei Technologies {Lu.Zhengdong,HangLi.HL}@huawei.com 3ADAPT Centre, School of Computing, Dublin City University Abstract The recently proposed neural network joint model (NNJM) (Devlin et al., 2014) augments the n-gram target lan- guage model with a heuristically cho- sen source context window, achieving state-of-the-art performance in SMT. In this paper, we give a more sys- tematic treatment by summarizing the relevant source information through a convolutional architecture guided by the target information. With dif- ferent guiding signals during decod- ing, our specifically designed convolu- tion+gating architectures can pinpoint the parts of a source sentence that are relevant to predicting a target word, and fuse them with the context of en- tire source sentence to form a unified representation. This representation, to- gether with target language words, are fed to a deep neural network (DNN) to form a stronger NNJM. Experiments on two NIST Chinese-English trans- lation tasks show that the proposed model can achieve significant improve- ments over the previous NNJM by up to +1.08 BLEU points on average. 1 Introduction Learning of continuous space representation for source language has attracted much at- tention in both traditional statistical machine translation (SMT) and neural machine trans- lation (NMT). Various models, mostly neural network-based, have been proposed for repre- senting the source sentence, mainly as the en- coder part in an encoder-decoder framework (Bengio et al., 2003; Auli et al., 2013; Kalch- brenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014). There has been some quite recent work on encoding only “relevant” part of source sentence during the decoding process, most notably neural network joint model (NNJM) in (Devlin et al., 2014), which extends the n-grams target language model by additionally taking a fixed-length window of source sentence, achieving state-of-the-art per- formance in statistical machine translation. In this paper, we propose novel convolu- tional architectures to dynamically encode the relevant information in the source language. Our model covers the entire source sentence, but can effectively find and properly summa- rize the relevant parts, guided by the informa- tion from the target language. With the guiding signals during decoding, our specifically de- signed convolution architectures can pinpoint the parts of a source sentence that are relevant to predicting a target word, and fuse them with the context of entire source sentence to form a unified representation. This representation, to- gether with target words, are fed to a deep neu- ral network (DNN) to form a stronger NNJM. Since our proposed joint model is purely lexi- calized, it can be integrated into any SMT de- coder as a feature. Two …

Content truncated to 500 words