Cross-lingual Similarity of Multilingual Representations Revisited

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 185–195 November 20–23, 2022. ©2022 Association for Computational Linguistics 185 Cross-lingual Similarity of Multilingual Representations Revisited Maksym Del and Mark Fishel Institute of Computer Science University of Tartu, Estonia {maksym,mark}@tartunlp.ai Abstract Related works used indexes like CKA and variants of CCA to measure the similarity of cross-lingual representations in multilingual language models. In this paper, we argue that assumptions of CKA/CCA align poorly with one of the motivating goals of cross-lingual learning analysis, i.e., explaining zero-shot cross-lingual transfer. We highlight what valu- able aspects of cross-lingual similarity these indexes fail to capture and provide a motivating case study demonstrating the problem empir- ically. Then, we introduce Average Neuron- Wise Correlation (ANC) as a straightforward al- ternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross- lingual context. Finally, we use ANC to con- struct evidence that the previously introduced “first align, then predict” pattern takes place not only in masked language models (MLMs) but also in multilingual models with causal lan- guage modeling objectives (CLMs). Moreover, we show that the pattern extends to the scaled versions of the MLMs and CLMs (up to 85x original mBERT).1 1 Introduction Similarity indexes like Canonical Correlation Anal- ysis (CCA, Hotelling, 1936) or Centered Kernel Alignment (CKA, Kornblith et al., 2019) aim to find a similarity between parallel sets of different representations of the same data. The deep learn- ing community adapted these indexes to measure similarity between representations that come from different models (Raghu et al., 2017; Morcos et al., 2018; Kornblith et al., 2019). Another line of work used the same methods to measure similarity be- tween different languages which come from a sin- gle multilingual model (Kudugunta et al., 2019; Singh et al., 2019a; Conneau et al., 2020; Muller et al., 2021). 1Our code is publicly available at https://github. com/TartuNLP/xsim In this paper, we argue that while CCA/CKA methods are a good fit for the first case, they are a suboptimal choice for the second scenario. First, we employ a real-world motivating exam- ple to demonstrate that CKA can fail to capture the notion of similarity that we consider helpful in a cross-lingual context. We also discuss the general problems of CKA/CCA indexes and conclude that they are not well aligned with some of the goals of cross-lingual analysis (Section 4). Next, we propose and verify an Averaged Neuron-Wise Correlation (ANC) as a straightfor- ward alternative. It exploits the fact that represen- tations from the same model have apriori-aligned neurons, which is the desired property in a cross- lingual setup (Section 5). Finally, Muller et al. (2021) demonstrated the so-called “first align, then predict” representational pattern in a multilingual model: the model first aligns representations of different languages to- gether, and then (starting from the middle layers) makes them more language-specific again (to ac- company …

Content truncated to 500 words