How Multilingual is Multilingual BERT?

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4996 How multilingual is Multilingual BERT? Telmo Pires∗ Eva Schlinger Dan Garrette Google Research {telmop,eschling,dhgarrette}@google.com Abstract In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2019) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annota- tions in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs. 1 Introduction Deep, contextualized language models provide powerful, general-purpose linguistic represen- tations that have enabled significant advances among a wide range of natural language process- ing tasks (Peters et al., 2018b; Devlin et al., 2019). These models can be pre-trained on large corpora of readily available unannotated text, and then fine-tuned for specific tasks on smaller amounts of supervised data, relying on the induced language model structure to facilitate generalization beyond the annotations. Previous work on model prob- ing has shown that these representations are able to encode, among other things, syntactic and named entity information, but they have heretofore fo- cused on what models trained on English capture about English (Peters et al., 2018a; Tenney et al., 2019b,a). ∗Google AI Resident. In this paper, we empirically investigate the degree to which these representations generalize across languages. We explore this question us- ing Multilingual BERT (henceforth, M-BERT), re- leased by Devlin et al. (2019) as a single language model pre-trained on the concatenation of mono- lingual Wikipedia corpora from 104 languages.1 M-BERT is particularly well suited to this probing study because it enables a very straightforward ap- proach to zero-shot cross-lingual model transfer: we fine-tune the model using task-specific super- vised training data from one language, and evalu- ate that task in a different language, thus allowing us to observe the ways in which the model gener- alizes information across languages. Our results show that M-BERT is able to perform cross-lingual generalization surprisingly well. More importantly, we present the results of a number of probing experiments designed to test various hypotheses about how the model is able to perform this transfer. Our experiments show that while high lexical overlap between languages im- proves transfer, M-BERT is also able to transfer between languages written in different scripts— thus having zero lexical overlap—indicating that it captures multilingual representations. We fur- ther show that transfer works best for typolog- ically similar languages, suggesting that while M-BERT’s multilingual representation is able to map learned structures onto new vocabularies, …

Content truncated to 500 words