Improving Neural Machine Translation with Neural Syntactic Distance

Improving Neural Machine Translation with Neural Syntactic Distance Chunpeng Ma1,3, Akihiro Tamura2, Masao Utiyama3, Tiejun Zhao1∗, Eiichiro Sumita3 1Harbin Institute of Technology, Harbin, China 2Ehime University, Matsuyama, Japan 3National Institute of Information and Communications Technology, Kyoto, Japan {cpma, tjzhao}@hit.edu.cn tamura@cs.ehime-u.ac.jp {mutiyama, eiichiro.sumita}@nict.go.jp Abstract The explicit use of syntactic information has been proved useful for neural machine trans- lation (NMT). However, previous methods re- sort to either tree-structured neural networks or long linearized sequences, both of which are inefficient. Neural syntactic distance (NSD) enables us to represent a constituent tree us- ing a sequence whose length is identical to the number of words in the sentence. NSD has been used for constituent parsing, but not in machine translation. We propose five strate- gies to improve NMT with NSD. Experiments show that it is not trivial to improve NMT with NSD; however, the proposed strategies are shown to improve translation performance of the baseline model (+2.1 (En–Ja), +1.3 (Ja– En), +1.2 (En–Ch), and +1.0 (Ch–En) BLEU). 1 Introduction In recent years, neural machine translation (NMT) has been developing rapidly and has become the de facto approach for machine translation. To im- prove the performance of the conventional NMT models (Sutskever et al., 2014; Bahdanau et al., 2014), one effective approach is to incorporate syntactic information into the encoder and/or de- coder of the baseline model. Based on how the syntactic information is represented, there are two categories of syn- tactic NMT methods: (1) those that use tree- structured neural networks (NNs) to represent syn- tax structures (Eriguchi et al., 2016; Hashimoto and Tsuruoka, 2017), and (2) those that use linear-structured NNs to represent linearized syn- tax structures (Li et al., 2017; Ma et al., 2017, 2018). For the first category, there is a direct corresponding relationship between the syntactic structure and the NN structure, but the complex- ity of NN structures usually makes training in- ∗Corresponding author efficient. In contrast, for the second category, syntactic structures are linearized and represented using linear-structured recurrent neural networks (RNNs), but the linearized sequence can generally be quite long and therefore training efficiency is still a problem. Although using a shorter sequence may improve the efficiency, some syntactic infor- mation is lost. We propose a method of using syntactic infor- mation in NMT that overcomes the disadvantages of both methods. The basis of our method is the neural syntactic distance (NSD), a recently pro- posed concept used for constituent parsing (Shen et al., 2018; G´omez-Rodr´ıguez and Vilares, 2018). NSD makes it possible to represent a constituent tree as a sequence whose length is identical to the number of words in the sentence (almost) without losing syntactic information. However, there are no previous studies that use NSD in NMT. More- over, as demonstrated by our experiments, using NSD in NMT is far from straightforward, so we propose five strategies and verify the effects em- pirically. The strategies are summarized below. • Extend NSD to dependency trees, which is inspired by the dependency language model (Shen et al., …

Content truncated to 500 words