Graph Convolutional Encoders for Syntax-aware Neural Machine Translation

Graph Convolutional Encoders for Syntax-aware Neural Machine Translation Jasmijn Bastings1 Ivan Titov1,2 Wilker Aziz1 Diego Marcheggiani1 Khalil Sima’an1 1ILLC, University of Amsterdam 2ILCC, University of Edinburgh {bastings,titov,w.aziz,marcheggiani,k.simaan}@uva.nl Abstract We present a simple and effective ap- proach to incorporating syntactic struc- ture into neural attention-based encoder- decoder models for machine translation. We rely on graph-convolutional networks (GCNs), a recent class of neural networks developed for modeling graph-structured data. Our GCNs use predicted syntactic dependency trees of source sentences to produce representations of words (i.e. hid- den states of the encoder) that are sensitive to their syntactic neighborhoods. GCNs take word representations as input and produce word representations as output, so they can easily be incorporated as layers into standard encoders (e.g., on top of bidi- rectional RNNs or convolutional neural networks). We evaluate their effectiveness with English-German and English-Czech translation experiments for different types of encoders and observe substantial im- provements over their syntax-agnostic ver- sions in all the considered setups. 1 Introduction Neural machine translation (NMT) is one of suc- cess stories of deep learning in natural language processing, with recent NMT systems outperform- ing traditional phrase-based approaches on many language pairs (Sennrich et al., 2016a). State-of- the-art NMT systems rely on sequential encoder- decoders (Sutskever et al., 2014; Bahdanau et al., 2015) and lack any explicit modeling of syntax or any hierarchical structure of language. One poten- tial reason for why we have not seen much benefit from using syntactic information in NMT is the lack of simple and effective methods for incorpo- rating structured information in neural encoders, including RNNs. Despite some successes, tech- niques explored so far either incorporate syntactic information in NMT models in a relatively indi- rect way (e.g., multi-task learning (Luong et al., 2015a; Nadejde et al., 2017; Eriguchi et al., 2017; Hashimoto and Tsuruoka, 2017)) or may be too restrictive in modeling the interface between syn- tax and the translation task (e.g., learning repre- sentations of linguistic phrases (Eriguchi et al., 2016)). Our goal is to provide the encoder with access to rich syntactic information but let it de- cide which aspects of syntax are beneficial for MT, without placing rigid constraints on the in- teraction between syntax and the translation task. This goal is in line with claims that rigid syntac- tic constraints typically hurt MT (Zollmann and Venugopal, 2006; Smith and Eisner, 2006; Chiang, 2010), and, though these claims have been made in the context of traditional MT systems, we believe they are no less valid for NMT. Attention-based NMT systems (Bahdanau et al., 2015; Luong et al., 2015b) represent source sen- tence words as latent-feature vectors in the en- coder and use these vectors when generating a translation. Our goal is to automatically incorpo- rate information about syntactic neighborhoods of source words into these feature vectors, and, thus, potentially improve quality of the translation out- put. Since vectors correspond to words, it is natu- ral for us to use dependency syntax. Dependency trees (see Figure 1) represent syntactic relations between …

Content truncated to 500 words