Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1936–1945 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1177 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1936–1945 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1177 Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder Huadong Chen†, Shujian Huang†∗, David Chiang‡, Jiajun Chen† †State Key Laboratory for Novel Software Technology, Nanjing University {chenhd,huangsj,chenjj}@nlp.nju.edu.cn ‡Department of Computer Science and Engineering, University of Notre Dame dchiang@nd.edu Abstract Most neural machine translation (NMT) models are based on the sequential encoder-decoder framework, which makes no use of syntactic information. In this pa- per, we improve this model by explicitly incorporating source-side syntactic trees. More specifically, we propose (1) a bidi- rectional tree encoder which learns both sequential and tree structured representa- tions; (2) a tree-coverage model that lets the attention depend on the source-side syntax. Experiments on Chinese-English translation demonstrate that our proposed models outperform the sequential atten- tional model as well as a stronger baseline with a bottom-up tree encoder and word coverage.1 1 Introduction Recently, neural machine translation (NMT) mod- els (Sutskever et al., 2014; Bahdanau et al., 2015) have obtained state-of-the-art performance on many language pairs. Their success depends on the representation they use to bridge the source and target language sentences. However, this rep- resentation, a sequence of fixed-dimensional vec- tors, differs considerably from most theories about mental representations of sentences, and from tra- ditional natural language processing pipelines, in which semantics is built up compositionally using a recursive syntactic structure. Perhaps as evidence of this, current NMT mod- els still suffer from syntactic errors such as at- tachment (Shi et al., 2016). We argue that instead of letting the NMT model rely solely on the im- plicit structure it learns during training (Cho et al., ∗ Corresponding author. 1Our code is publicly available at https://github. com/howardchenhd/Syntax-awared-NMT/ (a) example sentence pair with alignments aozhou x1 chongxin x2 kaifang x3 zhu x4 manila x5 dashiguan x6 (b) binarized source side tree Figure 1: An example sentence pair (a), with its binarized source side tree (b). We use xi to rep- resent the i-th word in the source sentence. We will use this sentence pairs as the running exam- ple throughout this paper. 2014a), we can improve its performance by aug- menting it with explicit structural information and using this information throughout the model. This has two benefits. First, the explicit syntactic information will help the encoder generate better source side represen- tations. Li et al. (2015) show that for tasks in which long-distance semantic dependencies mat- ter, representations learned from recursive mod- els using syntactic structures may be more pow- erful than those from sequential recurrent models. In the NMT case, given syntactic information, it will be easier for the encoder to incorporate long distance dependencies into better representations, which is especially important for the translation of long sentences. Second, …

Content truncated to 500 words