Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations

Proceedings of NAACL-HLT 2019, pages 1151–1161 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1151 Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations Meishan Zhang1 and Zhenghua Li2 and Guohong Fu3∗and Min Zhang2 1. School of New Media and Communication, Tianjin University, China 2. School of Computer Science and Technology, Soochow University, China 3. Institute of Artificial Intelligence, Soochow University, China mason.zms@gmail.com, {zhli13, minzhang}@suda.edu.cn, ghfu@hotmail.com Abstract Syntax has been demonstrated highly effec- tive in neural machine translation (NMT). Pre- vious NMT models integrate syntax by rep- resenting 1-best tree outputs from a well- trained parsing system, e.g., the representa- tive Tree-RNN and Tree-Linearization meth- ods, which may suffer from error propaga- tion. In this work, we propose a novel method to integrate source-side syntax implicitly for NMT. The basic idea is to use the interme- diate hidden representations of a well-trained end-to-end dependency parser, which are re- ferred to as syntax-aware word representations (SAWRs). Then, we simply concatenate such SAWRs with ordinary word embeddings to en- hance basic NMT models. The method can be straightforwardly integrated into the widely- used sequence-to-sequence (Seq2Seq) NMT models. We start with a representative RNN- based Seq2Seq baseline system, and test the effectiveness of our proposed method on two benchmark datasets of the Chinese-English and English-Vietnamese translation tasks, re- spectively. Experimental results show that the proposed approach is able to bring significant BLEU score improvements on the two datasets compared with the baseline, 1.74 points for Chinese-English translation and 0.80 point for English-Vietnamese translation, respectively. In addition, the approach also outperforms the explicit Tree-RNN and Tree-Linearization methods. 1 Introduction In the past few years, neural machine transla- tion (NMT) has drawn increasing interests due to its simplicity and promising performance (Bah- danau et al., 2014; Jean et al., 2015; Luong and Manning, 2015; Luong et al., 2015; Shen et al., 2016; Vaswani et al., 2017). The widely used ∗Corresponding author. 教育 是 现代 文明 的 基石 Education is modern civilization ’s cornerstone top root amod assmod assm attr • An example of input dependency tree. Input 教育 是 现代 文明 的 基石 Encoder o1 o2 o3 o4 o5 o6 Decoder head=1, top head=0, root head=4, amod head=6, assmod head=4, assm head=2, attr • SAWRs, where the encoder outputs are used as inputs for NMT similar to source-side word embeddings. Figure 1: An example to illustrate our method of en- coding source dependency syntax, where the English translation is “Education is the cornerstone of modern civilization” for the source Chinese input. sequence-to-sequence (Seq2Seq) framework com- bined with attention mechanism achieves signif- icant improvement over the traditional statistical machine translation (SMT) models on a variety of language pairs, such as Chinese-English (Shi et al., 2016; Mi et al., 2016; Vaswani et al., 2017; Cheng et al., 2018). Under an encoder-decoder ar- chitecture, the Seq2Seq framework first encodes the source sentence into a sequence of hidden vec- tors, and then incrementally predicts the target sentence (Cho et al., 2014a). Recently, inspired by the …

Content truncated to 500 words