Improving Sequence to Sequence Neural Machine Translation by Utilizing Syntactic Dependency Information

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 21–29, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Improving Sequence to Sequence Neural Machine Translation by Utilizing Syntactic Dependency Information An Nguyen Le, Ander Martinez, Akifumi Yoshimoto∗and Yuji Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology, 8916-5, Takayama, Ikoma, Nara 630-0192, Japan {nguyen.an.mr9,ander.martinez.zy4,akifumi-y,matsu}@is.naist.jp Abstract Sequence to Sequence Neural Machine Translation has achieved significant per- formance in recent years. Yet, there are some existing issues that Neural Machine Translation still does not solve completely. Two of them are translation of long sen- tences and “over-translation”. To address these two problems, we propose an ap- proach that utilize more grammatical in- formation such as syntactic dependencies, so that the output can be generated based on more abundant information. In addi- tion, the output of the model is presented not as a simple sequence of tokens but as a linearized tree construction. Experiments on the Europarl-v7 dataset of French-to- English translation demonstrate that our proposed method improves BLEU scores by 1.57 and 2.40 on datasets consisting of sentences with up to 50 and 80 tokens, respectively. Furthermore, the proposed method also solved the two existing prob- lems, ineffective translation of long sen- tences and over-translation in Neural Ma- chine Translation. 1 Introduction Our task is to construct a model which learns in- put in sequence form and decodes output as a lin- earized dependency tree. In this work, we propose an approach in which dependency labels are in- corporated into the model to represent more gram- matical information in the output sequence. As we know, the Sequence to Sequence (Seq2Seq) Learning model (Sutskever et al., 2014; Aha- roni et al., 2016) is extremely effective on a va- ∗This author’s present affiliation is CyberAgent, Inc., Tokyo, Japan, yoshimoto akifumi xa@cyberagent.co.jp riety of tasks that require a mapping between a se- quence to sequence. Therefore, it is used to solve many tasks in natural language processing. The Seq2Seq model consists of an encoder-decoder neural network which encodes a variable-length input sequence into a vector and decodes it into a variable-length output. Since the model uses the information of the source representation and the previously generated words to produce the next-word token, this distributed representation al- lows the Seq2Seq model to generate appropri- ate mapping between the input and the output (Li et al., 2016). For specific tasks, Neural Machine Translation (NMT) model, which is based on the Seq2Seq learning, has achieved excellent transla- tion performance in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Firat et al., 2016). In particular, the NMT model which is built upon an encoder-decoder frame- work with attention mechanism (Bahdanau et al., 2015) can also pay attention and its decoder knows which part of the input is relevant for the word that is currently being translated. Therefore, it has shown competitive results and outperformed conventional statistical methods (Bentivogli et al., 2016). Despite of these …

Content truncated to 500 words