Does String-Based Neural MT Learn Source Syntax?

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Does String-Based Neural MT Learn Source Syntax? Xing Shi, Inkit Padhi, and Kevin Knight Information Sciences Institute & Computer Science Department University of Southern California xingshi@isi.edu, ipadhi@usc.edu, knight@isi.edu Abstract We investigate whether a neural, encoder- decoder translation system learns syntactic in- formation on the source side as a by-product of training. We propose two methods to de- tect whether the encoder has learned local and global source syntax. A fine-grained analy- sis of the syntactic structure learned by the encoder reveals which kinds of syntax are learned and which are missing. 1 Introduction The sequence to sequence model (seq2seq) has been successfully applied to neural machine translation (NMT) (Sutskever et al., 2014; Cho et al., 2014) and can match or surpass MT state-of-art. Non- neural machine translation systems consist chiefly of phrase-based systems (Koehn et al., 2003) and syntax-based systems (Galley et al., 2004; Galley et al., 2006; DeNeefe et al., 2007; Liu et al., 2011; Cowan et al., 2006), the latter of which adds syntac- tic information to source side (tree-to-string), target side (string-to-tree) or both sides (tree-to-tree). As the seq2seq model first encodes the source sentence into a high-dimensional vector, then decodes into a target sentence, it is hard to understand and interpret what is going on inside such a procedure. Consider- ing the evolution of non-neural translation systems, it is natural to ask: 1. Does the encoder learn syntactic information about the source sentence? 2. What kind of syntactic information is learned, and how much? 3. Is it useful to augment the encoder with addi- tional syntactic information? In this work, we focus on the first two questions and propose two methods: • We create various syntactic labels of the source sentence and try to predict these syntactic la- bels with logistic regression, using the learned sentence encoding vectors (for sentence-level labels) or learned word-by-word hidden vectors (for word-level label). We find that the encoder captures both global and local syntactic infor- mation of the source sentence, and different in- formation tends to be stored at different layers. • We extract the whole constituency tree of source sentence from the NMT encoding vec- tors using a retrained linearized-tree decoder. A deep analysis on these parse trees indicates that much syntactic information is learned, while various types of syntactic information are still missing. 2 Example As a simple example, we train an English-French NMT system on 110M tokens of bilingual data (En- glish side). We then take 10K separate English sen- tences and label their voice as active or passive. We use the learned NMT encoder to convert these sen- tences into 10k corresponding 1000-dimension en- coding vectors. We use 9000 sentences to train a logistic regression model to predict voice using the encoding cell states, and test on the other 1000 sen- tences. We achieve 92.8% accuracy (Table 2), far above the majority …

Content truncated to 500 words