On the Multilingual Capabilities of Very Large-Scale English Language Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3056–3068 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3056 On the Multilingual Capabilities of Very Large-Scale English Language Models Jordi Armengol-Estap´e∗, Ona de Gibert Bonet∗, Maite Melero Barcelona Supercomputing Center Plac¸a Eusebi G¨uell 1-3, Barcelona 08034, Spain {jordi.armengol, ona.degibert, maite.melero}@bsc.es Abstract Generative Pre-trained Transformers (GPTs) have recently been scaled to unprecedented sizes in the history of machine learning. These language models have been shown to exhibit outstanding zero, one, and few-shot learning capabilities in a number of different tasks. Nevertheless, aside from anecdotal experiences, little is known regarding their multilingual capabilities, given the fact that the pre-training corpus is almost entirely composed of English text. In this work, we investigate its potential and limits in three tasks: extractive Question-Answering, text summarization and natural language generation for five different languages, as well as the effect of scale in terms of model size. Our results show that GPT-3 can be used, not only as a powerful generative pre-trained model for English, but for other languages as well, even for some with very few data in the training corpora, with room for improvement if optimization of the tokenization is addressed. Keywords: Multilingual, Cross-lingual, Language Modeling 1. Introduction Improving Natural Language Understanding (NLU) and Generation (NLG) by pre-training autoregressive language models based on the Transformer (Vaswani et al., 2017) decoder architecture has been common- place since the original GPT (Generative Pretrained Transformer) (Radford and Narasimhan, 2018) first ap- peared. In the race to scale up these language mod- els (Radford et al., 2019), the arrival of GPT-3 (Brown et al., 2020) has changed the rules of the game. As claimed by their creators, its ability to learn from a few examples “via text interaction” makes it stand out from the rest. Its impressive generative capabilities have caused a big sensation, not only at research level but also in the mainstream media. A particular feature of GPT-3 is, besides the sheer size of the data it has been trained on, the fact that, al- though the data is generally of good quality, it has not been filtered for language (in purpose). Therefore, al- though GPT-3 is in principle a language model for En- glish, its training data contains many other languages,1 even if they account for a small portion of the dataset in comparison to English (93% by word count). Intu- itively, one would expect that this quantity would not be enough to obtain a high-quality language model in these other languages, especially in the low-resource ones. Some evidence in this regard is provided by the large amount of data required to train language- specific models (Nozza et al., 2020). Even the multi- lingual ones2 such as mBERT (Devlin et al., 2018) or *Equal contribution. 1https://github.com/openai/gpt-3/tree/ master/dataset_statistics 2Note that both mBERT and XLM-R are encoder-based models, unlike GPT, but the point still holds. XLM-R (Conneau et al., 2019) employ large multilin- gual datasets based on Wikipedia …

Content truncated to 500 words