Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5877–5891 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review Fred Philippy1,2∗and Siwen Guo1 and Shohreh Haddadan1 1Zortify Labs, Zortify S.A. 19, rue du Laboratoire L-1911 Luxembourg 2SnT, University of Luxembourg 29, Avenue J.F Kennedy L-1359 Luxembourg {fred, siwen, shohreh}@zortify.com Abstract In recent years, pre-trained Multilingual Lan- guage Models (MLLMs) have shown a strong ability to transfer knowledge across different languages. However, given that the aspiration for such an ability has not been explicitly in- corporated in the design of the majority of MLLMs, it is challenging to obtain a unique and straightforward explanation for its emer- gence. In this review paper, we survey literature that investigates different factors contributing to the capacity of MLLMs to perform zero-shot cross-lingual transfer and subsequently outline and discuss these factors in detail. To enhance the structure of this review and to facilitate con- solidation with future studies, we identify five categories of such factors. In addition to provid- ing a summary of empirical evidence from past studies, we identify consensuses among studies with consistent findings and resolve conflicts among contradictory ones. Our work contex- tualizes and unifies existing research streams which aim at explaining the cross-lingual po- tential of MLLMs. This review provides, first, an aligned reference point for future research and, second, guidance for a better-informed and more efficient way of leveraging the cross- lingual capacity of MLLMs. 1 Introduction The objective of cross-lingual transfer is to leverage knowledge learned by a model in a source language and to transfer it to a target language. While such a process of transferring knowledge and concepts across languages seems natural for a polyglot, it is believed to be less straightforward for a language model. Nevertheless, multilingual language mod- els (MLLMs), such as mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019) and XLM-R (Conneau et al., 2020a) demonstrate effective cross- lingual transfer capabilities. Such a transfer ability is moderately expected from XLM, given that par- ∗Research was conducted at Zortify. allel data is leveraged through a cross-lingual trans- fer learning objective during pre-training. How- ever, it is less anticipated for mBERT and XLM- R, which are pre-trained on separate monolingual corpora without any explicit cross-lingual signal. Nevertheless, the latter show a surprisingly strong cross-lingual transfer capacity on a variety of down- stream tasks (Hu et al., 2020). While no apparent factors explaining the nature of this ability can be intuitively derived from the properties of MLLMs, there have been many attempts to understand this behavior. Past research has outlined and investi- gated various factors that may impact cross-lingual transfer performance in MLLMs, but there are still open questions due to conflicting findings across studies. In our work, we inspect findings from past research investigating the inner workings of cross- lingual transfer in MLLMs. We not only outline overlapping contributions with consensual …

Content truncated to 500 words