Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563 - 3574 December 7-11, 2022 ©2022 Association for Computational Linguistics Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models Terra Blevins1 Luke Zettlemoyer1,2 1 Paul G. Allen School of Computer Science & Engineering, University of Washington 2 Meta AI Research {blvns, lsz}@cs.washington.edu Abstract English pretrained language models, which make up the backbone of many modern NLP systems, require huge amounts of unlabeled training data. These models are generally pre- sented as being trained only on English text but have been found to transfer surprisingly well to other languages. We investigate this phenomenon and find that common English pretraining corpora actually contain significant amounts of non-English text: even when less than 1% of data is not English (well within the error rate of strong language classifiers), this leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percent- ages of non-English data facilitate cross-lingual transfer for models trained on them, with tar- get language performance strongly correlated to the amount of in-language data seen during pretraining. In light of these findings, we ar- gue that no model is truly monolingual when pretrained at scale, which should be considered when evaluating cross-lingual transfer. 1 Introduction Pretrained language models have become an in- tegral part of NLP systems. They come in two flavors: monolingual, where the model is trained on text from a single language, and multilingual, where the model is jointly trained on data from many different languages. Monolingual pretrained models are generally applied to tasks in the same language, whereas multilingual ones are used for cross-lingual tasks or transfer. Recent work has claimed that monolingual pretrained models are also surprisingly good at transferring between languages, despite ostensi- bly having never seen the target language before (Gogoulou et al., 2021; Li et al., 2021, inter alia). However, because of the large scale of pretrain- ing data and because many pretraining corpora are not publicly available, it is currently unknown how Figure 1: Estimated non-English data in English pre- training corpora (token count and total percentage); even small percentages lead to many tokens. C4.En (†) is es- timated from the first 50M examples in the corpus. much foreign language data exists in monolingual pretraining corpora. In this paper, we show that (1) these data are almost certainly contaminated with very small percentages of text from other languages and that (2) cross-lingual transfer is possible from such data leakage in the pretraining corpus. More specifically, we quantify how multilingual English pretrained models are in two steps. First, we analyze common English pretraining corpora with a large-scale automatic evaluation to estimate their language composition, as well as a smaller- scale manual analysis. Second, we perform ex- periments across fifty languages on masked lan- guage modeling and part-of-speech (POS) tagging to measure how well the models trained on these pretraining corpora perform outside of English. Our analysis finds that these corpora …

Content truncated to 500 words