Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 1–10, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks Yonatan Belinkov1 Llu´ıs M`arquez2 Hassan Sajjad2 Nadir Durrani2 Fahim Dalvi2 James Glass1 1MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA 02139, USA {belinkov, glass}@mit.edu 2Qatar Computing Research Institute, HBKU, Doha, Qatar {lmarquez, hsajjad, ndurrani, faimaduddin}@qf.org.qa Abstract While neural machine translation (NMT) models provide improved translation qual- ity in an elegant framework, it is less clear what they learn about language. Re- cent work has started evaluating the qual- ity of vector representations learned by NMT models on morphological and syn- tactic tasks. In this paper, we investigate the representations learned at different lay- ers of NMT encoders. We train NMT sys- tems on parallel data and use the mod- els to extract features for training a clas- sifier on two tasks: part-of-speech and se- mantic tagging. We then measure the per- formance of the classifier as a proxy to the quality of the original NMT model for the given task. Our quantitative analysis yields interesting insights regarding repre- sentation learning in NMT models. For in- stance, we find that higher layers are bet- ter at learning semantics while lower lay- ers tend to be better for part-of-speech tag- ging. We also observe little effect of the target language on source-side representa- tions, especially in higher quality models.1 1 Introduction Neural machine translation (NMT) offers an el- egant end-to-end architecture, while at the same time improving translation quality. However, little is known about the inner workings of these models and their interpretability is limited. Recent work has started exploring what kind of linguistic infor- mation such models learn on morphological (Vy- lomova et al., 2016; Belinkov et al., 2017; Dalvi et al., 2017) and syntactic levels (Shi et al., 2016; Sennrich, 2017). 1Our code is available at http://github.com/ boknilev/nmt-repr-analysis. One observation that has been made is that lower layers in the neural MT network learn different kinds of information than higher lay- ers. For example, Shi et al. (2016) and Belinkov et al. (2017) found that representations from lower layers of the NMT encoder are more predictive of word-level linguistic properties like part-of- speech (POS) and morphological tags, whereas higher layer representations are more predictive of more global syntactic information. In this work, we take a first step towards understanding what NMT models learn about semantics. We evaluate NMT representations from different layers on a se- mantic tagging task and compare to the results on a POS tagging task. We believe that understand- ing the semantics learned in NMT can facilitate using semantic information for improving NMT systems, as previously shown for non-neural MT (Chan et al., 2007; Liu and Gildea, 2010; Gao and Vogel, 2011; Wu et al., 2011; Jones et al., 2012; Bazrafshan and Gildea, 2013, 2014). For the semantic (SEM) tagging task, we use the dataset recently introduced by …

Content truncated to 500 words