Wiki-40B: Multilingual Language Model Dataset

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2440–2452 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2440 Wiki-40B: Multilingual Language Model Dataset Mandy Guo, Zihang Dai, Denny Vrandeˇci´c, Rami Al-Rfou Google Research 1600 Amphitheatre Parkway, Mountain View CA {xyguo, zihangd, vrandecic, rmyeid}@google.com Abstract We propose a new multilingual language model benchmark that is composed of 40+ languages spanning several scripts and linguistic families. With around 40 billion characters, we hope this new resource will accelerate the research of multilingual modeling. We train monolingual causal language models using a state-of-the-art model (Transformer-XL) establishing baselines for many languages. We also introduce the task of multilingual causal language modeling where we train our model on the combined text of 40+ languages from Wikipedia with different vocabulary sizes and evaluate on the languages individually. We released the cleaned-up text of 40+ Wikipedia language editions, the corresponding trained monolingual language models, and several multilingual language models with different fixed vocabulary sizes. Keywords: Language Modeling, Wikipedia, Multilinguality, Low Resource Languages 1. Introduction Language modeling has received a significant attention for its role as a benchmark to test new network architectures (Vaswani et al., 2017; Dai et al., 2019; Al-Rfou et al., 2019) and learning representations for down-stream tasks (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2018; Yang et al., 2019). Causal language models are a category of language models that aim to predict the next token given the previously observed tokens. They have been deployed in a wide variety of practical applications such as machine translation, automatic speech recognition, spelling correc- tion and writing assistants (Kannan et al., 2016; Henderson et al., 2017; Chen et al., 2019). Language model evaluation metrics such as perplexity and bits per character are intrinsic in nature. They enable re- searchers to use this task to evaluate different networks and training algorithms robustly. Extrinsic evaluation, on the other hand, relies on down-stream tasks which are usu- ally small or quite limited in scope, if they exist at all for low resource languages. Causal language model evaluation datasets can be scaled easily to avoid robustness issues that come with small down-stream evaluation tasks. Despite all these benefits, most of the released datasets with benchmarked results are limited to English (Prasad et al., 2008; Mahoney, 2009; Chelba et al., 2013). This limits the ability of researchers to understand many aspects of language modeling that are related to open vocabulary and complex morphology given that compiling a small vocab- ulary for English can already achieve a high coverage rate (Baayen, 1996; Kageura, 2012). It is uncommon for researchers to study causal language modeling within a multilingual corpus that contains a mix- ture of languages. However, with the rising interests in mul- tilingual research, multilingual language modeling could be of great interest to researchers in the fields of “Multi- Domain” or “Multi-Task” modeling. We hope that this novel setup will pose new challenges for researchers. We aim to transfer knowledge …

Content truncated to 500 words