Cross-lingual Transfer of Monolingual Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 948–955 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 948 Cross-lingual Transfer of Monolingual Models Evangelia Gogoulou⋆, Ariel Ekgren◦, Tim Isbister◦, Magnus Sahlgren◦ ⋆RISE Research Institutes of Sweden, ◦AI Sweden evangelia.gogoulou@ri.se {ariel.ekgren, tim.isbister, magnus.sahlgren}@ai.se Abstract Recent studies in cross-lingual learning using multilingual models have cast doubt on the previous hypothesis that shared vocabulary and joint pre-training are the keys to cross-lingual generalization. We introduce a method for transferring monolingual models to other languages through continuous pre-training and study the effects of such transfer from four different languages to English. Our experimental results on GLUE show that the transferred models outperform an English model trained from scratch, independently of the source language. After probing the model representations, we find that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in English. 1. Introduction Training a language model from scratch requires consid- erable resources, both with respect to data and computa- tional resources. These requirements can be a limiting factor for many actors, and for smaller languages, it is not clear whether there even exists enough data to train a language model. Consequently, there is an increasing interest in using cross-lingual transfer to alleviate the requirements of training a language model from scratch. Most of the work in this direction (see the next section) concerns multilingual models. In this paper, we want to explore the effects of cross- lingual transfer of a monolingual model into a target language space. More specifically, we want to inves- tigate if it is possible to adapt existing monolingual models to the target language and study their down- stream performance in comparison with a model trained from scratch in the target language. Additionally, we want to study the impact of language similarity between source and target language on fine-tuning performance after transfer. Based on recent work that shows that transformer-based language models encode universal properties (Lu et al., 2021), we hypothesize that model knowledge learned in the source language enhances the learning of the target language independently of lan- guage proximity. Our contributions in this work are the following: (i) we introduce an adaptation method for cross-lingual trans- fer (Section 3.), (ii) we show that the models that have been transferred to English outperform an En- glish model trained from scratch in the GLUE bench- mark for all source languages studied here (Section 4.2.), (iii) through probing the model representations, we demonstrate that abstractions learned in the source language are transferred to English (Section 4.3.). 2. Related Work The standard methodology of transferring knowledge across languages is by training either cross-lingual (Ruder et al., 2019) or multilingual models (Schwenk and Douze, 2017; Devlin et al., 2019; Conneau et al., 2020a; Xue et al., 2020; Chung et al., 2021). These latter models are trained on massively multilingual data using a shared vocabulary, which has proven to be successful for zero-shot cross-lingual transfer (Pires et al., …

Content truncated to 500 words