Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10725 - 10742 December 7-11, 2022 ©2022 Association for Computational Linguistics Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models Fabian David Schmidt1, Ivan Vuli´c2,3, Goran Glavaš1 1 Center For Artificial Intelligence and Data Science, University of Würzburg, Germany 2 Language Technology Lab, University of Cambridge, UK 3 PolyAI Ltd., UK {fabian.schmidt, goran.glavas}@uni-wuerzburg.de iv250@cam.ac.uk Abstract A large body of recent work highlights the fal- lacies of zero-shot cross-lingual transfer (ZS- XLT) with large multilingual language models. Namely, their performance varies substantially for different target languages and is the weakest where needed the most: for low-resource lan- guages distant to the source language. One rem- edy is few-shot transfer (FS-XLT), where lever- aging only a few task-annotated instances in the target language(s) may yield sizable perfor- mance gains. However, FS-XLT also succumbs to large variation, as models easily overfit to the small datasets. In this work, we present a sys- tematic study focused on a spectrum of FS-XLT fine-tuning regimes, analyzing key properties such as effectiveness, (in)stability, and modular- ity. We conduct extensive experiments on both higher-level (NLI, paraphrasing) and lower- level tasks (NER, POS), presenting new FS-XLT strategies that yield both improved and more stable FS-XLT across the board. Our findings challenge established FS-XLT methods: e.g., we propose to replace sequential fine-tuning with joint fine-tuning on source and target lan- guage instances, offering consistent gains with different number of shots (including resource- rich scenarios). We also show that further gains can be achieved with multi-stage FS-XLT train- ing in which joint multilingual fine-tuning pre- cedes the bilingual source-target specialization. 1 Introduction and Motivation Successful fine-tuning of mainstream pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Conneau et al., 2020) for various NLP tasks requires a sizeable set of labeled task-specific in- stances. While such abundant task data are avail- able for many tasks in English and a few high- resource languages, annotated examples are much scarcer for low-resource languages (Joshi et al., 2020). A large body of recent work thus focused on zero-shot cross-lingual transfer (ZS-XLT), for which no labeled instances are available in the tar- get language (Pires et al., 2019; Cao et al., 2020). Catalyzed by pretrained massively multilingual transformers (MMT) such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), or mT5 (Xue et al., 2021), ZS-XLT has achieved impressive results on a wide variety of tasks (Hu et al., 2020; Ruder et al., 2021). The MMT-driven ZS-XLT, how- ever, exhibits dramatic performance drops when transferring to low-resource languages and/or lan- guages distant from the source language (Lauscher et al., 2020; Ebrahimi et al., 2021; Adelani et al., 2021, inter alia). In contrast, recent work high- lights that language models are excellent few-shot learners (Brown et al., 2020; Gao et al., 2021): they adapt well to new tasks or languages when exposed to only on a handful of labeled instances. For cross-lingual transfer in …

Content truncated to 500 words