Sequence-to-Dependency Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 698–707 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1065 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 698–707 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1065 Sequence-to-Dependency Neural Machine Translation Shuangzhi Wu†∗, Dongdong Zhang‡ , Nan Yang‡ , Mu Li‡ , Ming Zhou‡ †Harbin Institute of Technology, Harbin, China ‡Microsoft Research {v-shuawu, dozhang, nanya, muli, mingzhou}@microsoft.com Abstract Nowadays a typical Neural Machine Translation (NMT) model generates trans- lations from left to right as a linear sequence, during which latent syntactic structures of the target sentences are not explicitly concerned. Inspired by the suc- cess of using syntactic knowledge of tar- get language for improving statistical ma- chine translation, in this paper we propose a novel Sequence-to-Dependency Neural Machine Translation (SD-NMT) method, in which the target word sequence and its corresponding dependency structure are jointly constructed and modeled, and this structure is used as context to facilitate word generations. Experimental results show that the proposed method signifi- cantly outperforms state-of-the-art base- lines on Chinese-English and Japanese- English translation tasks. 1 Introduction Recently, Neural Machine Translation (NMT) with the attention-based encoder-decoder frame- work (Bahdanau et al., 2015) has achieved sig- nificant improvements in translation quality of many language pairs (Bahdanau et al., 2015; Lu- ong et al., 2015a; Tu et al., 2016; Wu et al., 2016). In a conventional NMT model, an encoder reads in source sentences of various lengths, and trans- forms them into sequences of intermediate hidden vector representations. After weighted by atten- tion operations, combined hidden vectors are used by the decoder to generate translations. In most of cases, both encoder and decoder are implemented as recurrent neural networks (RNNs). ∗Contribution during internship at Microsoft Research. Many methods have been proposed to further improve the sequence-to-sequence NMT model since it was first proposed by Sutskever et al. (2014) and Bahdanau et al. (2015). Previous work ranges from addressing the problem of out-of- vocabulary words (Jean et al., 2015), designing at- tention mechanism (Luong et al., 2015a), to more efficient parameter learning (Shen et al., 2016), using source-side syntactic trees for better encod- ing (Eriguchi et al., 2016) and so on. All these NMT models employ a sequential recurrent neu- ral network for target generations. Although in theory RNN is able to remember sufficiently long history, we still observe substantial incorrect trans- lations which violate long-distance syntactic con- straints. This suggests that it is still very challeng- ing for a linear RNN to learn models that effec- tively capture many subtle long-range word de- pendencies. For example, Figure 1 shows an in- correct translation related to the long-distance de- pendency. The translation fragment in italic is lo- cally fluent around the word is, but from a global view the translation is ungrammatical. Actually, this part of translation should be mostly affected by the distant plural noun foreigners rather than words …

Content truncated to 500 words