What do Neural Machine Translation Models Learn about Morphology?

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 861–872 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1080 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 861–872 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1080 What do Neural Machine Translation Models Learn about Morphology? Yonatan Belinkov1 Nadir Durrani2 Fahim Dalvi2 Hassan Sajjad2 James Glass1 1MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA 02139, USA {belinkov, glass}@mit.edu 2Qatar Computing Research Institute, HBKU, Doha, Qatar {ndurrani, faimaduddin, hsajjad}@qf.org.qa Abstract Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architec- ture. However, little is known about what these models learn about source and tar- get languages during the training process. In this work, we analyze the representa- tions learned by neural MT models at var- ious levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks. We conduct a thorough investiga- tion along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations. Our data-driven, quanti- tative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.1 1 Introduction Neural network models are quickly becoming the predominant approach to machine translation (MT). Training neural MT (NMT) models can be done in an end-to-end fashion, which is sim- pler and more elegant than traditional MT sys- tems. Moreover, NMT systems have become competitive with, or better than, the previous state-of-the-art, especially since the introduction of sequence-to-sequence models and the atten- tion mechanism (Bahdanau et al., 2014; Sutskever et al., 2014). The improved translation quality is often attributed to better handling of non-local dependencies and morphology generation (Luong 1Our code is available at https://github.com/ boknilev/nmt-repr-analysis. and Manning, 2015; Bentivogli et al., 2016; Toral and S´anchez-Cartagena, 2017). However, little is known about what and how much these models learn about each language and its features. Recent work has started ex- ploring the role of the NMT encoder in learn- ing source syntax (Shi et al., 2016), but research studies are yet to answer important questions such as: (i) what do NMT models learn about word morphology? (ii) what is the effect on learning when translating into/from morphologically-rich languages? (iii) what impact do different repre- sentations (character vs. word) have on learning? and (iv) what do different modules learn about the syntactic and semantic structure of a language? Answering such questions is imperative for fully understanding the NMT architecture. In this pa- per, we strive towards exploring (i), (ii), and (iii) by providing quantitative, data-driven answers to the following specific questions: • Which parts of the NMT architecture capture word structure? • What is the division of labor between differ- ent components (e.g. different layers or en- coder vs. decoder)? • How do different word representations help learn better morphology …

Content truncated to 500 words