On the Cross-lingual Transferability of Monolingual Representations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4623 On the Cross-lingual Transferability of Monolingual Representations Mikel Artetxe†∗, Sebastian Ruder‡ , Dani Yogatama‡ †HiTZ Center, University of the Basque Country (UPV/EHU) ‡DeepMind mikel.artetxe@ehu.eus {ruder,dyogatama}@google.com Abstract State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross- lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that trans- fers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective—freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators. 1 Introduction Multilingual pre-training methods such as multi- lingual BERT (mBERT, Devlin et al., 2019) have been successfully used for zero-shot cross-lingual transfer (Pires et al., 2019; Conneau and Lample, 2019). These methods work by jointly training a ∗Work done as an intern at DeepMind. transformer model (Vaswani et al., 2017) to per- form masked language modeling (MLM) in multi- ple languages, which is then fine-tuned on a down- stream task using labeled data in a single language— typically English. As a result of the multilingual pre-training, the model is able to generalize to other languages, even if it has never seen labeled data in those languages. Such a cross-lingual general- ization ability is surprising, as there is no explicit cross-lingual term in the underlying training objec- tive. In relation to this, Pires et al. (2019) hypothe- sized that: . . .having word pieces used in all languages (num- bers, URLs, etc), which have to be mapped to a shared space forces the co-occurring pieces to also be mapped to a shared space, thus spreading the ef- fect to other word pieces, until different languages are close to a shared space. ...mBERT’s ability to generalize cannot be at- tributed solely to vocabulary memorization, and that it must be learning a deeper multilingual rep- resentation. Cao et al. (2020) echoed this sentiment, and Wu and Dredze (2019) further observed that mBERT performs better in languages that share many sub- words. As such, the current consensus of the cross- lingual generalization ability of mBERT is …

Content truncated to 500 words