Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1500 - 1512 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability Yoshinari Fujinuma∗ AWS AI Labs Amazon.com fujinumay@gmail.com Jordan Boyd-Graber UMIACS, CS, LSC, iSchool University of Maryland jbg@umiacs.umd.edu Katharina Kann Computer Science University of Colorado Boulder katharina.kann@colorado.edu Abstract Pretrained multilingual models enable zero- shot learning even for unseen languages, and that performance can be further improved via adaptation prior to finetuning. However, it is unclear how the number of pretraining lan- guages influences a model’s zero-shot learn- ing for languages unseen during pretraining. To fill this gap, we ask the following re- search questions: (1) How does the number of pretraining languages influence zero-shot performance on unseen target languages? (2) Does the answer to that question change with model adaptation? (3) Do the findings for our first question change if the languages used for pretraining are all related? Our experi- ments on pretraining with related languages indicate that choosing a diverse set of lan- guages is crucial. Without model adaptation, surprisingly, increasing the number of pre- training languages yields better results up to adding related languages, after which perfor- mance plateaus. In contrast, with model adap- tation via continued pretraining, pretraining on a larger number of languages often gives fur- ther improvement, suggesting that model adap- tation is crucial to exploit additional pretrain- ing languages.1 1 Introduction Pretrained multilingual language models (Devlin et al., 2019; Conneau et al., 2020) are now a stan- dard approach for cross-lingual transfer in natural language processing (NLP). However, there are multiple, potentially related issues on pretraining multilingual models. Conneau et al. (2020) find the “curse of multilinguality”: for a fixed model size, zero-shot performance on target languages seen during pretraining increases with additional pre- training languages only until a certain point, after ∗This work was done while the first author was a student at University of Colorado Boulder. 1All code used in this paper is available at https: //github.com/akkikiki/multilingual_ zeroshot_analysis. which performance decreases. Wang et al. (2020b) also report “negative interference”, where monolin- gual models achieve better results than multilingual models, both on subsets of high- and low-resource languages. However, those findings are limited to target languages seen during pretraining. Current multilingual models cover only a small subset of the world’s languages. Furthermore, due to data sparsity, monolingual pretrained models are not likely to obtain good results for many low- resource languages. In those cases, multilingual models can zero-shot learn for unseen languages with an above-chance performance, which can be further improved via model adaptation with target- language text (Wang et al., 2020a), even for limited amounts (Ebrahimi and Kann, 2021). However, it is poorly understood how the number of pretraining languages influences performance in those cases. Does the “curse of multilinguality” or “negative interference” also impact performance on unseen target languages? And, if we want a model to be applicable to …

Content truncated to 500 words