From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4483–4499, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4483 From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers Anne Lauscher1∗, Vinit Ravishankar2∗, Ivan Vuli´c3, and Goran Glavaˇs1 1Data and Web Science Group, University of Mannheim, Germany 2Language Technology Group, University of Oslo, Norway 3Language Technology Lab, University of Cambridge, UK 1{anne,goran}@informatik.uni-mannheim.de, 2vinitr@ifi.uio.no, 3iv250@cam.ac.uk Abstract Massively multilingual transformers (MMTs) pretrained via language modeling (e.g., mBERT, XLM-R) have become a default paradigm for zero-shot language transfer in NLP, offering unmatched transfer perfor- mance. Current evaluations, however, verify their efficacy in transfers (a) to languages with sufficiently large pretraining corpora, and (b) between close languages. In this work, we an- alyze the limitations of downstream language transfer with MMTs, showing that, much like cross-lingual word embeddings, they are substantially less effective in resource-lean scenarios and for distant languages. Our experiments, encompassing three lower-level tasks (POS tagging, dependency parsing, NER) and two high-level tasks (NLI, QA), em- pirically correlate transfer performance with linguistic proximity between source and target languages, but also with the size of target language corpora used in MMT pretraining. Most importantly, we demonstrate that the inexpensive few-shot transfer (i.e., additional fine-tuning on a few target-language instances) is surprisingly effective across the board, war- ranting more research efforts reaching beyond the limiting zero-shot conditions. 1 Introduction and Motivation Labeled datasets of sufficient size support super- vised learning in NLP. The notorious tediousness, subjectivity, and cost of linguistic annotation (Dan- dapat et al., 2009; Sabou et al., 2012; Fort, 2016), coupled with plethora of structurally different NLP tasks, lead to existence of such datasets only for a handful of resource-rich languages (Bender, 2011; Ponti et al., 2019; Joshi et al., 2020). This data scarcity renders the need for effective cross-lingual transfer strategies: how can we exploit abundant labeled data from resource-rich languages to make ∗Equal contribution. predictions in resource-lean languages? In the most extreme scenario, termed zero-shot cross-lingual transfer, not a single labeled instance exists for a target language. Recent work has placed much emphasis on this scenario exactly; in theory, it of- fers the widest portability across the world’s 7,000+ languages (Pires et al., 2019; Artetxe et al., 2020b; Lin et al., 2019; Cao et al., 2020; Hu et al., 2020). The current mainstay of cross-lingual transfer in NLP are approaches based on continuous cross- lingual representation spaces such as cross-lingual word embeddings (CLWEs) (Ruder et al., 2019) and, most recently, massively multilingual trans- former networks (MMTs), pretrained on multilin- gual corpora with language modeling (LM) ob- jectives (Devlin et al., 2019; Conneau and Lam- ple, 2019; Conneau et al., 2020). The latter have de facto become the default language transfer paradigm, with multiple studies reporting their un- paralleled transfer performance (Pires et al., 2019; Wu and Dredze, 2019; R¨onnqvist et al., 2019; Karthikeyan et al., 2020; Wu et al., 2020). Key Questions and Contributions. In this work, we dissect the current state-of-the-art …

Content truncated to 500 words