Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3575 - 3590 December 7-11, 2022 ©2022 Association for Computational Linguistics Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models Terra Blevins1 Hila Gonen1,2 Luke Zettlemoyer1,2 1 Paul G. Allen School of Computer Science & Engineering, University of Washington 2 Meta AI Research {blvns, lsz}@cs.washington.edu hilagnn@gmail.com Abstract The emergent cross-lingual transfer seen in multilingual pretrained models has sparked significant interest in studying their behavior. However, because these analyses have focused on fully trained multilingual models, little is known about the dynamics of the multilingual pretraining process. We investigate when these models acquire their in-language and cross- lingual abilities by probing checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language per- formance early on, with lower-level linguis- tic skills acquired before more complex ones. In contrast, the point in pretraining when the model learns to transfer cross-lingually differs across language pairs. Interestingly, we also observe that, across many languages and tasks, the final model layer exhibits significant perfor- mance degradation over time, while linguistic knowledge propagates to lower layers of the network. Taken together, these insights high- light the complexity of multilingual pretraining and the resulting varied behavior for different languages over time. 1 Introduction Large-scale language models pretrained jointly on text from many different languages (Delvin, 2019; Lample and Conneau, 2019; Lin et al., 2021) per- form very well on various languages and on cross- lingual transfer between them (e.g., Kondratyuk and Straka, 2019; Pasini et al., 2021). Due to this success, there has been a great deal of interest in uncovering what these models learn from the mul- tilingual pretraining signal (§6). However, these works analyze a single model artifact: the final training checkpoint at which the model is consid- ered to be converged. Recent work has also studied monolingual models by expanding the analysis to multiple pretraining checkpoints to see how model knowledge changes across time (Liu et al., 2021). We analyze multilingual training checkpoints throughout the pretraining process in order to identify when multilingual models obtain their in- language and cross-lingual abilities. The case of multilingual language models is particularly inter- esting, as the model learns both to capture individ- ual languages and to transfer between them just from unbalanced multitask language modeling for each language. Specifically, we retrain a popular multilingual model, XLM-R (Conneau et al., 2020a), and run a suite of linguistic tasks covering 59 languages on checkpoints from across the pretraining process.1 This suite evaluates different syntactic and seman- tic skills in both monolingual and cross-lingual transfer settings. While our analysis primarily fo- cuses on the knowledge captured in model output representations over time, we also consider how the performance of internal layers changes during pretraining for a subset of tasks. Our analysis uncovers several insights into mul- tilingual knowledge acquisition. First, while the model acquires most in-language linguistic infor- mation early on, cross-lingual transfer is learned …

Content truncated to 500 words