WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3992 - 4006 July 10-15, 2022 ©2022 Association for Computational Linguistics WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models Benjamin Minixhofer1 and Fabian Paischer2,3 and Navid Rekabsaz1,3 1Institute of Computational Perception, Johannes Kepler University Linz 2Institute for Machine Learning, Johannes Kepler University Linz 3ELLIS Unit Linz and LIT AI Lab {benjamin.minixhofer, navid.rekabsaz}@jku.at paischer@ml.jku.at Abstract Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models re- quires ever more computational resources and most of the existing models are trained on En- glish text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method – called WECHSEL – to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target lan- guage and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the tar- get language. We use WECHSEL to trans- fer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outper- forms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available. 1 Introduction Large LMs based on the Transformer architec- ture (Vaswani et al., 2017) have become increas- ingly popular since GPT (Radford et al., 2018) and BERT (Devlin et al., 2019) were introduced, prompting the creation of many large LMs pre- trained on English text (Yang et al., 2019; Clark et al., 2020; Lewis et al., 2020; Ram et al., 2021). There is a tendency towards training larger and larger models (Brown et al., 2020; Fedus et al., 2021) while the main focus is on the English lan- guage. Recent work has called attention to the costs associated with training increasingly large LMs, in- cluding environmental and financial cost (Strubell et al., 2019; Bender et al., 2021). If training large LMs for English is already costly, it is prohibitively expensive to train new, similarly powerful models to cover other languages. One approach to address this issue is creating massively multilingual models (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021) trained on a concatenation of texts in many different languages. These models show strong natural language under- standing capabilities in a wide variety of languages, but suffer from what Conneau et al. (2020) call the curse of multilinguality: beyond …

Content truncated to 500 words