As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846 August 1–6, 2021. ©2021 Association for Computational Linguistics 836 As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages Wietse de Vries University of Groningen The Netherlands wietse.de.vries@rug.nl Malvina Nissim University of Groningen The Netherlands m.nissim@rug.nl Abstract Large generative language models have been very successful for English, but other lan- guages lag behind, in part due to data and com- putational limitations. We propose a method that may overcome these problems by adapt- ing existing pre-trained models to new lan- guages. Specifically, we describe the adapta- tion of English GPT-2 to Italian and Dutch by retraining lexical embeddings without tuning the Transformer layers. As a result, we obtain lexical embeddings for Italian and Dutch that are aligned with the original English lexical embeddings. Additionally, we scale up com- plexity by transforming relearned lexical em- beddings of GPT-2 small to the GPT-2 medium embedding space. This method minimises the amount of training and prevents losing infor- mation during adaptation that was learned by GPT-2. English GPT-2 models with relearned lexical embeddings can generate realistic sen- tences in Italian and Dutch. Though on aver- age these sentences are still identifiable as arti- ficial by humans, they are assessed on par with sentences generated by a GPT-2 model fully trained from scratch. 1 Introduction Large pre-trained language models have brought unprecedented progress in NLP, but also concerns regarding the excessive computing power needed to train them (Strubell et al., 2019). Limited access to large amounts of computational resources, as well as environmental considerations, curb possi- bilities for less-resourced and less-researched lan- guages. Additionally, models like GPT-2 (Radford et al., 2019) are trained on amounts of data that are not available for most languages. As a result of these limitations, language models are commonly trained for English, whereas reproductions in other languages may underperform or not exist. That language models can benefit from informa- tion in other languages has been demonstrated by the effectiveness of multilingual BERT (mBERT) and XLM-RoBERTa (Conneau et al., 2020). How- ever, for downstream tasks mBERT has been shown to be outperformed by monolingual models for higher resource languages whereas lower resource languages can still achieve better results without pre-trained language models (Nozza et al., 2020; Wu and Dredze, 2020). Rather than pursuing a multilingual direction, we aim at exploiting existing language models and language similarities to create models for new languages. Specifically, we develop a multi-step procedure for adapting English GPT-2 (Radford et al., 2019) to Italian and Dutch. Dutch is geneti- cally closely related to English, both being West- Germanic languages, while Italian is a more distant Romance language from the same Indo-European language family (Eberhard et al., 2020). It is how- ever worth noticing that at sentence level English and Italian tend to have the same word order (SVO), while Dutch is SVO in main clauses, but SOV in subordinate ones; at noun phrase level, English and Dutch share constituent …

Content truncated to 500 words