A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4903 - 4915 July 10-15, 2022 ©2022 Association for Computational Linguistics A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank Dan Malkin♢ Tomasz Limisiewicz♠∗ Gabriel Stanovsky♢ ♢ School of Computer Science, The Hebrew University of Jerusalem ♠ Faculty of Mathematics and Physics, Charles University in Prague {dan.malkinhueb,gabriel.stanovsky}@mail.huji.ac.il limisiewicz@ufal.mff.cuni.cz Abstract We show that the choice of pretraining lan- guages affects downstream cross-lingual trans- fer for BERT-based models. We inspect zero- shot performance in balanced data conditions to mitigate data size confounds, classifying pre- training languages that improve downstream performance as donors, and languages that are improved in zero-shot performance as recipi- ents. We develop a method of quadratic time complexity in the number of languages to esti- mate these relations, instead of an exponential exhaustive computation of all possible combi- nations. We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. Our findings can inform developers of large- scale multilingual language models in choosing better pretraining configurations.1 1 Introduction Pretrained language models are setting state-of-the- art results by leveraging raw texts during pretrain- ing (PLMs; Peters et al., 2018; Devlin et al., 2019, inter alia). Interestingly, when pretraining on mul- tilingual corpora, PLMs seem to exhibit zero-shot cross-lingual abilities, achieving non-trivial perfor- mance on downstream examples in languages seen only during pretraining. For example, in the bot- tom of Figure 1, a named entity recognition model finetuned on Russian is capable of predicting cor- rectly name entity tags for texts in English, seen only during pretraining (Pires et al., 2019; Conneau et al., 2020b; K et al., 2020; Conneau et al., 2020a; Lazar et al., 2021; Turc et al., 2021). Previous analyses examined how several factors contribute to this emerging behavior. For example, parameter sharing and model depth are important in certain configurations (K et al., 2020; Conneau et al., 2020b), as well as typological similarities ∗Work done while visiting the Hebrew University. 1Code and models are publicly available at: github. com/SLAB-NLP/linguistic-blood-bank Figure 1: We build a complete, directed graph over a diverse set of 22 languages. Weighted edges show the improvement of bilingual LM over monolingual performance (bold edges represent larger weights). Lan- guages which consistently improve performance are termed “donors” and marked in red, while languages which benefit most are termed “recipients” (marked in blue). We show that our observations hold in several configurations on two downstream tasks. between languages (Pires et al., 2019), and the choice of specific finetune languages (Turc et al., 2021). In this work, we focus on an important factor that we find missing in prior work, namely the effect that pretraining languages have on down- stream zero-shot performance. In particular, we ask three major research questions: (1) Does the choice 4903 of pretraining languages affect downstream cross- lingual transfer, and if so, to what extent? (2) Is English the …

Content truncated to 500 words