Emerging Cross-lingual Structure in Pretrained Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6022 Emerging Cross-lingual Structure in Pretrained Language Models Alexis Conneau♥∗ Shijie Wu♠∗ Haoran Li♥ Luke Zettlemoyer♥ Veselin Stoyanov♥ ♠Department of Computer Science, Johns Hopkins University ♥Facebook AI aconneau@fb.com, shijie.wu@jhu.edu {aimeeli,lsz,ves}@fb.com Abstract We study the problem of multilingual masked language modeling, i.e. the training of a sin- gle model on concatenated text from multi- ple languages, and present a detailed study of several factors that influence why these mod- els are so effective for cross-lingual transfer. We show, contrary to what was previously hy- pothesized, that transfer is possible even when there is no shared vocabulary across the mono- lingual corpora and also when the text comes from very different domains. The only require- ment is that there are some shared parameters in the top layers of the multi-lingual encoder. To better understand this result, we also show that representations from monolingual BERT models in different languages can be aligned post-hoc quite effectively, strongly suggesting that, much like for non-contextual word em- beddings, there are universal latent symme- tries in the learned embedding spaces. For multilingual masked language modeling, these symmetries are automatically discovered and aligned during the joint training process. 1 Introduction Multilingual language models such as mBERT (De- vlin et al., 2019) and XLM (Lample and Conneau, 2019) enable effective cross-lingual transfer — it is possible to learn a model from supervised data in one language and apply it to another with no additional training. Recent work has shown that transfer is effective for a wide range of tasks (Wu and Dredze, 2019; Pires et al., 2019). These work speculates why multilingual pretraining works (e.g. shared vocabulary), but only experiment with a single reference mBERT and is unable to systemat- ically measure these effects. In this paper, we present the first detailed em- pirical study of the effects of different masked lan- ∗Equal contribution. Work done while Shijie was intern- ing at Facebook AI. guage modeling (MLM) pretraining regimes on cross-lingual transfer. Our first set of experiments is a detailed ablation study on a range of zero-shot cross-lingual transfer tasks. Much to our surprise, we discover that language universal representations emerge in pretrained models without the require- ment of any shared vocabulary or domain similarity, and even when only a subset of the parameters in the joint encoder are shared. In particular, by sys- tematically varying the amount of shared vocabu- lary between two languages during pretraining, we show that the amount of overlap only accounts for a few points of performance in transfer tasks, much less than might be expected. By sharing parameters alone, pretraining learns to map similar words and sentences to similar hidden representations. To better understand these effects, we also ana- lyze multiple monolingual BERT models trained independently. We find that monolingual models trained in different languages learn representations that align with each other surprisingly well, even though they have no …

Content truncated to 500 words