Zero-Shot Cross-Lingual Transfer with Meta Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4547–4562, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4547 Zero-Shot Cross-Lingual Transfer with Meta Learning Farhad Nooralahzadeh University of Oslo farhad.nooralahzadeh@uzh.ch Giannis Bekoulis Vrije Universiteit Brussel - imec gbekouli@etrovub.be Johannes Bjerva University of Copenhagen / Aalborg University jbjerva@cs.aau.dk Isabelle Augenstein University of Copenhagen augenstein@di.ku.dk Abstract Learning what to share between tasks has be- come a topic of great importance, as strategic sharing of knowledge has been shown to im- prove downstream task performance. This is particularly important for multilingual appli- cations, as most languages in the world are under-resourced. Here, we consider the set- ting of training models on multiple different languages at the same time, when little or no data is available for languages other than En- glish. We show that this challenging setup can be approached using meta-learning: in addition to training a source language model, another model learns to select which train- ing instances are the most beneficial to the first. We experiment using standard super- vised, zero-shot cross-lingual, as well as few- shot cross-lingual settings for different natu- ral language understanding tasks (natural lan- guage inference, question answering). Our ex- tensive experimental setup demonstrates the consistent effectiveness of meta-learning for a total of 15 languages. We improve upon the state-of-the-art for zero-shot and few-shot NLI (on MultiNLI and XNLI) and QA (on the MLQA dataset). A comprehensive error analysis indicates that the correlation of typo- logical features between languages can partly explain when parameter sharing learned via meta-learning is beneficial. 1 Introduction There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each (Eberhard et al., 2019). De- spite this, very few languages have proper linguistic resources when it comes to natural language under- standing tasks (Joshi et al., 2020). Although there is growing awareness in the field, as evidenced by the release of datasets such as XNLI (Conneau et al., 2018), most NLP research still only consid- ers English (Bender, 2019). While one solution to this issue is to collect annotated data for all lan- guages, this process is both too time-consuming and expensive to be feasible. Additionally, it is not trivial to train a model for a task in a particular language (e.g., English) and apply it directly to an- other language where only limited training data is available (i.e., low-resource languages). Therefore, it is essential to investigate strategies that allow one to use the large amount of training data available for English for the benefit of other languages. Meta-learning has recently been shown to be beneficial for several machine learning tasks (Koch et al., 2015; Vinyals et al., 2016; Santoro et al., 2016; Finn et al., 2017; Ravi and Larochelle, 2017; Nichol et al., 2018). For NLP, recent work has also shown the benefits of this sharing between tasks and domains (Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019). Although cross-lingual trans- fer with …

Content truncated to 500 words