Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 833–844, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 833 Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT Shijie Wu and Mark Dredze Department of Computer Science Johns Hopkins University shijie.wu@jhu.edu, mdredze@cs.jhu.edu Abstract Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2019) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pre- trained on 104 languages with impressive per- formance for zero-shot cross-lingual transfer on a natural language inference task. This pa- per explores the broader cross-lingual poten- tial of mBERT (multilingual) as a zero-shot language transfer model on 5 NLP tasks cov- ering a total of 39 languages from various lan- guage families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effec- tive strategy for utilizing mBERT in this man- ner, determine to what extent mBERT general- izes away from language-specific features, and measure factors that influence cross-lingual transfer. 1 Introduction Pretrained language representations with self- supervised objectives have become standard in a variety of NLP tasks (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019), including sentence-level classifica- tion (Wang et al., 2018), sequence tagging (e.g. NER) (Tjong Kim Sang and De Meulder, 2003) and SQuAD question answering (Rajpurkar et al., 2016). Self-supervised objectives include lan- guage modeling, the cloze task (Taylor, 1953) and next sentence classification. These objectives con- tinue key ideas in word embedding objectives like CBOW and skip-gram (Mikolov et al., 2013a). Code is available at https://github.com/ shijie-wu/crosslingual-nlp At the same time, cross-lingual embedding mod- els have reduced the amount of cross-lingual su- pervision required to produce reasonable models; Conneau et al. (2017); Artetxe et al. (2018) use identical strings between languages as a pseudo bilingual dictionary to learn a mapping between monolingual-trained embeddings. Can jointly train- ing contextual embedding models over multiple languages without explicit mappings produce an effective cross-lingual representation? Surpris- ingly, the answer is (partially) yes. BERT, a re- cently introduced pretrained model (Devlin et al., 2019), offers a multilingual model (mBERT) pre- trained on concatenated Wikipedia data for 104 languages without any cross-lingual alignment (De- vlin, 2018). mBERT does surprisingly well com- pared to cross-lingual word embeddings on zero- shot cross-lingual transfer in XNLI (Conneau et al., 2018), a natural language inference dataset. Zero- shot cross-lingual transfer, also known as single- source transfer, refers trains and selects a model in a source language, often a high resource language, then transfers directly to a target language. While XNLI results are promising, the ques- tion remains: does mBERT learn a cross-lingual space that supports zero-shot transfer? We eval- uate mBERT as a zero-shot cross-lingual transfer model on five different NLP tasks: …

Content truncated to 500 words