Unsupervised Cross-lingual Representation Learning at Scale

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8440 Unsupervised Cross-lingual Representation Learning at Scale Alexis Conneau∗ Kartikay Khandelwal∗ Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzm´an Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov Facebook AI Abstract This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross- lingual transfer tasks. We train a Transformer- based masked language model on one hundred languages, using more than two terabytes of fil- tered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accu- racy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource lan- guages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previ- ous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and ca- pacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per- language performance; XLM-R is very compet- itive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.1 1 Introduction The goal of this paper is to improve cross-lingual language understanding (XLU), by carefully study- ing the effects of training unsupervised cross- lingual representations at a very large scale. We present XLM-R a transformer-based multilingual masked language model pre-trained on text in 100 languages, which obtains state-of-the-art perfor- mance on cross-lingual classification, sequence la- beling and question answering. ∗Equal contribution. Correspondence to {aconneau,kartikayk}@fb.com 1https://github.com/facebookresearch/(fairseq-py,pytext,xlm) Multilingual masked language models (MLM) like mBERT (Devlin et al., 2018) and XLM (Lam- ple and Conneau, 2019) have pushed the state- of-the-art on cross-lingual understanding tasks by jointly pretraining large Transformer mod- els (Vaswani et al., 2017) on many languages. These models allow for effective cross-lingual transfer, as seen in a number of benchmarks in- cluding cross-lingual natural language inference (Bowman et al., 2015; Williams et al., 2017; Con- neau et al., 2018), question answering (Rajpurkar et al., 2016; Lewis et al., 2019), and named en- tity recognition (Pires et al., 2019; Wu and Dredze, 2019). However, all of these studies pre-train on Wikipedia, which provides a relatively limited scale especially for lower resource languages. In this paper, we first present a comprehensive analysis of the trade-offs and limitations of multi- lingual language models at scale, inspired by re- cent monolingual scaling efforts (Liu et al., 2019). We measure the trade-off between high-resource and low-resource languages and the impact of lan- guage sampling and vocabulary size. The experi- ments expose a trade-off as we scale the number of languages for a fixed model capacity: more lan- guages leads to better cross-lingual performance …

Content truncated to 500 words