Neural Machine Translation with Source Dependency Representation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2846–2852 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Neural Machine Translation with Source Dependency Representation Kehai Chen1∗, Rui Wang2†, Masao Utiyama2, Lemao Liu3, Akihiro Tamura4, Eiichiro Sumita2 and Tiejun Zhao1 1Machine Intelligence & Translation Laboratory, Harbin Institute of Technology 2ASTREC, National Institute of Information and Communications Technology (NICT) 3Tencent AI Lab 4Graduate School of Science and Engineering, Ehime University {khchen and tjzhao}@hit.edu.cn {wangrui, mutiyama and eiichiro.sumita}@nict.go.jp lemaoliu@gmail.com and tamura@cs.ehime-u.ac.jp Abstract Source dependency information has been successfully introduced into statistical machine translation. However, there are only a few preliminary attempts for Neural Machine Translation (NMT), such as concatenating representations of source word and its dependency label together. In this paper, we propose a novel attentional NMT with source dependency represen- tation to improve translation performance of NMT, especially on long sentences. Empirical results on NIST Chinese-to- English translation task show that our method achieves 1.6 BLEU improvements on average over a strong NMT system. 1 Introduction Neural Machine Translation (NMT) (Kalch- brenner and Blunsom, 2013; Bahdanau et al., 2014; Sutskever et al., 2014) relies heavily on source representations, which encode implicitly semantic information of source words by neural networks (Mikolov et al., 2013a,b). Recently, several research works have been proposed to learn richer source representation, such as multi- source information (Zoph and Knight, 2016; Firat et al., 2016), and particularly source syntactic information (Eriguchi et al., 2016; Li et al., 2017; Huadong et al., 2017; Eriguchi et al., 2017), thus improving the performance of NMT. In this paper, we enhance source representations by dependency information, which can capture source long-distance dependency constraints for word prediction. Actually, source dependency information has been shown greatly effective in ∗Kehai Chen was an internship research fellow at NICT when conducting this work. †Corresponding author. Statistical Machine Translation (SMT) (Garmash and Monz, 2014; Kazemi et al., 2015; Hadiwinoto et al., 2016; Chen et al., 2017; Hadiwinoto and Ng, 2017). In NMT, there has been a quite recent preliminary exploration (Sennrich and Haddow, 2016), in which vector representations of source word and its dependency label are simply concatenated as source input, achieving state-of- the-art performance in NMT (Bojar et al., 2016). In this paper, we propose a novel NMT with source dependency representation to improve translation performance. Compared with the simple approach of vector concatenation, we learn the Source Dependency Representation (SDR) to compute dependency context vectors and alignment matrices in a more sophisticated manner, which has the potential to make full use of source dependency information. To this end, we create a dependency unit for each source word to capture long-distance dependency constraints. Then we design an Encoder with convolutional architecture to jointly learn SDRs (Section 3) and source dependency annotations, thus computing dependency context vectors and hidden states by a novel double-context based Decoder for word prediction (Section 4). Empirical results on NIST Chinese-to-English translation task show that the proposed approach achieves significant gains over the method by Sennrich …

Content truncated to 500 words