Discovering Language-neutral Sub-networks in Multilingual Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7560 - 7575 December 7-11, 2022 ©2022 Association for Computational Linguistics Discovering Language-neutral Sub-networks in Multilingual Language Models Negar Foroutan Mohammadreza Banaei Rémi Lebret Antoine Bosselut Karl Aberer {firstname.lastname}@epfl.ch EPFL Abstract Multilingual pre-trained language models trans- fer remarkably well on cross-lingual down- stream tasks. However, the extent to which they learn language-neutral representations (i.e., shared representations that encode similar phenomena across languages), and the effect of such representations on cross-lingual transfer performance, remain open questions. In this work, we conceptualize language neu- trality of multilingual models as a function of the overlap between language-encoding sub- networks of these models. We employ the lot- tery ticket hypothesis to discover sub-networks that are individually optimized for various lan- guages and tasks. Our evaluation across three distinct tasks and eleven typologically-diverse languages demonstrates that sub-networks for different languages are topologically similar (i.e., language-neutral), making them effective initializations for cross-lingual transfer with limited performance degradation.1 1 Introduction Multilingual language models (MultiLMs) such as mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R (Conneau et al., 2020a) are pre-trained jointly on raw data from mul- tiple languages. Later, when they are fine-tuned for a task using perhaps a single high-resource lan- guage dataset, they demonstrate promising zero- shot cross-lingual performance, generalizing to the same task in different languages despite not having been fine-tuned for those languages. The facilitator of this cross-lingual transfer abil- ity is often hypothesized to be a learned language neutrality in these models (i.e., similar linguistic phenomena across languages are represented sim- ilarly by the model). However, the source of this ability remains an open question. Certain studies 1Our code is available at https://github.com/negar- foroutan/multiLMs-lang-neutral-subnets. Urdu French (a) Step 1: Discover sub-networks in multilingual language models that encode particular languages French Urdu French Urdu ✓ ✓ ✓ (b) Step 2: Transfer sub-networks to new lan- guages to evaluate their language neutrality degree Figure 1: Overview of our approach. We discover sub- networks in the original multilingual language model that are good foundations for learning various tasks and languages (a). Then, we investigate to what extent these sub-networks are similar by transferring them across other task-language pairs (b). In this example, the blue and red lines show sub-networks found for French and Urdu, respectively, and purple connections are shared in both sub-networks. Dashed lines show the weights that are removed in the pruning phase. have investigated mBERT (a prominent MultiLM) for universal language-neutral components that would facilitate cross-lingual transfer (Libovický et al., 2019; Pires et al., 2019; Libovický et al., 2020). Meanwhile, other studies claim mBERT is not language-neutral as it partitions its multilingual semantic space into separate language-specific sub- spaces (Singh et al., 2019; Choenni and Shutova, 2020; Wu and Dredze, 2019). However, most of these prior works analyze the language neutrality 7560 of MultiLMs by probing their output contextual representations. In this paper, language-neutrality is instead con- ceptualized in terms of the parameters of Multi- …

Content truncated to 500 words