Investigating Multilingual NMT Representations at Scale

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1565–1575, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1565 Investigating Multilingual NMT Representations at Scale Sneha Kudugunta Ankur Bapna Isaac Caswell Orhan Firat Google AI {snehark,ankurbpn,icaswell,orhanf}@google.com Abstract Multilingual Neural Machine Translation (NMT) models have yielded large empirical success in transfer learning settings. However, these black-box representations are poorly understood, and their mode of transfer remains elusive. In this work, we attempt to understand massively multilingual NMT representations (with 103 languages) using Singular Value Canonical Correlation Analysis (SVCCA), a representation similarity framework that allows us to compare representations across different languages, layers and models. Our analysis validates several empirical results and long-standing intuitions, and unveils new observations regarding how representations evolve in a multilingual translation model. We draw three major conclusions from our analysis, with implications on cross-lingual transfer learning: (i) Encoder representations of different languages cluster based on linguistic similarity, (ii) Representations of a source language learned by the encoder are dependent on the target language, and vice-versa, and (iii) Representations of high resource and/or linguistically similar lan- guages are more robust when fine-tuning on an arbitrary language pair, which is critical to determining how much cross-lingual transfer can be expected in a zero or few-shot setting. We further connect our findings with existing empirical observations in multilingual NMT and transfer learning. 1 Introduction Multilingual Neural Machine Translation (NMT) models have demonstrated great improvements for cross-lingual transfer, on tasks including low- resource language translation (Zoph et al., 2016; Nguyen and Chiang, 2017; Neubig and Hu, 2018) and zero or few-shot transfer learning for down- stream tasks (Eriguchi et al., 2018; Lample and Conneau, 2019; Wu and Dredze, 2019). A possi- ble explanation is the ability of multilingual mod- els to encode text from different languages in a shared representation space, resulting in simi- lar sentences being aligned together (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Arivazhagan et al., 2019b). This is justified by the success of multilingual representations on tasks like sentence alignment across languages (Artetxe and Schwenk, 2018), zero-shot cross-lingual clas- sification (Eriguchi et al., 2018) and XNLI (Lam- ple and Conneau, 2019). Although there exist em- pirical results that suggest that transfer is stronger across similar languages (Zoph et al., 2016; Neu- big and Hu, 2018; Wu and Dredze, 2019), the mechanism of generalization in multilingual rep- resentations is poorly understood. While interpretability is still a nascent field, there has been some work on investigating the learning dynamics and nature of representations learnt by neural models (Olah et al., 2018). Singular Value Canonical Correlation Analysis (SVCCA) is one such method that allows us to analyze the similarity between noisy, high- dimensional representations of the same data- points learnt across different models, layers and tasks (Raghu et al., 2017). SVCCA has been used to understand the learning dynamics and represen- tational similarity in a variety of computer …

Content truncated to 500 words