Modeling Source Syntax for Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 688–697 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1064 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 688–697 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1064 Modeling Source Syntax for Neural Machine Translation Junhui Li† Deyi Xiong† Zhaopeng Tu‡∗ Muhua Zhu‡ Min Zhang† Guodong Zhou† †School of Computer Science and Technology, Soochow University, Suzhou, China {lijunhui, dyxiong, minzhang, gdzhou}@suda.edu.cn ‡Tencent AI Lab, Shenzhen, China tuzhaopeng@gmail.com, muhuazhu@tencent.com Abstract Even though a linguistics-free sequence to sequence model in neural machine trans- lation (NMT) has certain capability of im- plicitly learning syntactic information of source sentences, this paper shows that source syntax can be explicitly incorpo- rated into NMT effectively to provide fur- ther improvements. Specifically, we lin- earize parse trees of source sentences to obtain structural label sequences. On the basis, we propose three different sorts of encoders to incorporate source syntax into NMT: 1) Parallel RNN encoder that learns word and label annotation vectors paral- lelly; 2) Hierarchical RNN encoder that learns word and label annotation vectors in a two-level hierarchy; and 3) Mixed RNN encoder that stitchingly learns word and label annotation vectors over sequences where words and labels are mixed. Exper- imentation on Chinese-to-English transla- tion demonstrates that all the three pro- posed syntactic encoders are able to im- prove translation accuracy. It is interesting to note that the simplest RNN encoder, i.e., Mixed RNN encoder yields the best perfor- mance with an significant improvement of 1.4 BLEU points. Moreover, an in-depth analysis from several perspectives is pro- vided to reveal how source syntax benefits NMT. 1 Introduction Recently the sequence to sequence model (seq2seq) in neural machine translation (NMT) has achieved certain success over the state-of- the-art of statistical machine translation (SMT) ∗Work done at Huawei Noah’s Ark Lab, HongKong. �� ��� �� �� �� �� �� � NP2 NP1 VV tokoyo stock exchange approves new listing bank input: output: reference: tokyo exchange approves shinsei bank 's application for listing (a). An example of discontinuous translation �� �� � � �� , �� � � �� �� �� � NP they came from six families with two girls and two girls . they came from six families and two girls are without parents . (b). An example of over translation input: output: reference: Figure 1: Examples of NMT translation that fail to respect source syntax. on various language pairs (Bahdanau et al., 2015; Jean et al., 2015; Luong et al., 2015; Luong and Manning, 2015). However, Shi et al. (2016) show that the seq2seq model still fails to capture a lot of deep structural details, even though it is capable of learning certain implicit source syntax from sentence-aligned parallel corpus. Moreover, it requires an additional parsing-task-specific training mechanism to recover the hidden syntax in NMT. As a result, in the absence of explicit linguistic knowledge, the seq2seq model …

Content truncated to 500 words