Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2832–2841 August 1–6, 2021. ©2021 Association for Computational Linguistics 2832 Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads? Zae Myung Kim1,2, Laurent Besacier1, Vassilina Nikoulina1, Didier Schwab2 1NAVER LABS Europe 2Univ. Grenoble Alpes, CNRS, LIG {zae-myung.kim,laurent.besacier,vassilina.nikoulina}@naverlabs.com didier.schwab@univ-grenoble-alpes.fr Abstract Recent studies on the analysis of the mul- tilingual representations focus on identifying whether there is an emergence of language- independent representations, or whether a mul- tilingual model partitions its weights among different languages. While most of such work has been conducted in a “black-box” manner, this paper aims to analyze individual com- ponents of a multilingual neural translation (NMT) model. In particular, we look at the encoder self-attention and encoder-decoder at- tention heads (in a many-to-one NMT model) that are more specific to the translation of a cer- tain language pair than others by (1) employ- ing metrics that quantify some aspects of the attention weights such as “variance” or “con- fidence”, and (2) systematically ranking the importance of attention heads with respect to translation quality. Experimental results show that surprisingly, the set of most important at- tention heads are very similar across the lan- guage pairs and that it is possible to remove nearly one-third of the less important heads without hurting the translation quality greatly. 1 Introduction Recent work on analyzing the internals of Transformer-based models (Vaswani et al., 2017) sheds some light on how different components within the models affect the final performance (Bo- goychev, 2020; Behnke and Heafield, 2020), and are closely related to playing linguistically inter- pretable roles (Voita et al., 2019; Jo and Myaeng, 2020). Moreover, studies on the analysis of mul- tilingual representations (Conneau et al., 2020b; Dufter and Sch¨utze, 2020; Wang et al., 2020b) fo- cus on identifying whether there is an emergence of language-independent representations in mul- tilingual models, or whether multilingual models partition their weights among different languages. In this paper, we investigate if similar analy- sis can be made for pretrained multilingual neu- ral machine translation (NMT) models regarding language pair specificity. More precisely, we analyze multi-head attention in a many-to-one (Transformer-based) NMT model and try to find, through an extensive ablation method on selection of the attention heads, whether some heads are more specific to the translation of a certain lan- guage pair than others. Our contributions are the following: (1) we ex- amine the effectiveness of different attention-based metrics on pruning encoder self-attention and cross attention heads; (2) we find that while it is possible to discover rare heads that are specific to a language pair by using a proposed head selection method, most important heads are language-independent; (3) we also show that around 30% of heads can be removed with very little loss of performance. 2 Related Work Recent studies analyzed the roles of attention heads in the Transformer models either in language mod- eling (LM) (Michel et al., 2019; Clark et al., 2019; Jo and Myaeng, 2020) or NMT (Voita et al., …

Content truncated to 500 words