Neural Machine Translation with Source-Side Latent Graph Parsing

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 125–135 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Neural Machine Translation with Source-Side Latent Graph Parsing Kazuma Hashimoto and Yoshimasa Tsuruoka The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan {hassy,tsuruoka}@logos.t.u-tokyo.ac.jp Abstract This paper presents a novel neural ma- chine translation model which jointly learns translation and source-side latent graph representations of sentences. Un- like existing pipelined approaches using syntactic parsers, our end-to-end model learns a latent graph parser as part of the encoder of an attention-based neu- ral machine translation model, and thus the parser is optimized according to the translation objective. In experiments, we first show that our model compares favor- ably with state-of-the-art sequential and pipelined syntax-based NMT models. We also show that the performance of our model can be further improved by pre- training it with a small amount of tree- bank annotations. Our final ensemble model significantly outperforms the previ- ous best models on the standard English- to-Japanese translation dataset. 1 Introduction Neural Machine Translation (NMT) is an active area of research due to its outstanding empiri- cal results (Bahdanau et al., 2015; Luong et al., 2015; Sutskever et al., 2014). Most of the exist- ing NMT models treat each sentence as a sequence of tokens, but recent studies suggest that syntac- tic information can help improve translation accu- racy (Eriguchi et al., 2016b, 2017; Sennrich and Haddow, 2016; Stahlberg et al., 2016). The exist- ing syntax-based NMT models employ a syntactic parser trained by supervised learning in advance, and hence the parser is not adapted to the transla- tion tasks. An alternative approach for leveraging syntactic structure in a language processing task is to jointly learn syntactic trees of the sentences All the calculated electronic band structures are metallic . 0.42 0.34 0.20 0.44 0.34 0.71 0.27 0.45 Figure 1: An example of the learned latent graphs. Edges with a small weight are omitted. along with the target task (Socher et al., 2011; Yo- gatama et al., 2017). Motivated by the promising results of recent joint learning approaches, we present a novel NMT model that can learn a task-specific latent graph structure for each source-side sentence. The graph structure is similar to the dependency struc- ture of the sentence, but it can have cycles and is learned specifically for the translation task. Un- like the aforementioned approach of learning sin- gle syntactic trees, our latent graphs are composed of “soft” connections, i.e., the edges have real- valued weights (Figure 1). Our model consists of two parts: one is a task-independent parsing com- ponent, which we call a latent graph parser, and the other is an attention-based NMT model. The latent parser can be independently pre-trained with human-annotated treebanks and is then adapted to the translation task. In experiments, we demonstrate that our model can be effectively pre-trained by the treebank annotations, outperforming a state-of-the-art se- quential counterpart and a pipelined syntax-based model. Our final ensemble model outperforms the …

Content truncated to 500 words