Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5712–5730 July 9-14, 2023 ©2023 Association for Computational Linguistics Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging Fabian David Schmidt1, Ivan Vuli´c2, Goran Glavaš1 1 Center For Artificial Intelligence and Data Science, University of Würzburg, Germany 2 Language Technology Lab, University of Cambridge, UK {fabian.schmidt, goran.glavas}@uni-wuerzburg.de iv250@cam.ac.uk Abstract Massively multilingual language models have displayed strong performance in zero-shot (ZS- XLT) and few-shot (FS-XLT) cross-lingual trans- fer setups, where models fine-tuned on task data in a source language are transferred with- out any or with only a few annotated instances to the target language(s). However, current work typically overestimates model perfor- mance as fine-tuned models are frequently eval- uated at model checkpoints that generalize best to validation instances in the target languages. This effectively violates the main assumptions of ‘true’ ZS-XLT and FS-XLT. Such XLT se- tups require robust methods that do not depend on labeled target language data for validation and model selection. In this work, aiming to improve the robustness of ‘true’ ZS-XLT and FS- XLT, we propose a simple and effective method that averages different checkpoints (i.e., model snapshots) during task fine-tuning. We conduct exhaustive ZS-XLT and FS-XLT experiments across higher-level semantic tasks (NLI, extrac- tive QA) and lower-level token classification tasks (NER, POS). The results indicate that averaging model checkpoints yields system- atic and consistent performance gains across diverse target languages in all tasks. Impor- tantly, it simultaneously substantially desensi- tizes XLT to varying hyperparameter choices in the absence of target language validation. We also show that checkpoint averaging bene- fits performance when further combined with run averaging (i.e., averaging the parameters of models fine-tuned over independent runs). 1 Introduction and Motivation Massively multilingual transformers (MMT) such as mBERT (Devlin et al., 2019) and XLM-R (Con- neau et al., 2020) have become the main driver of multilingual NLP research. When fine-tuned on sizable task data in a high-resource source lan- guage, typically English, MMTs demonstrate cross- lingual transfer capabilities (Pires et al., 2019) in zero-shot (ZS-XLT; without any task-annotated in- stances in the target language) and few-shot (FS- XLT; only a few task-annotated instances/shots available in the target language) transfer setups (Hu et al., 2020; Lauscher et al., 2020). However, re- cent work has shown that both cross-lingual trans- fer (XLT) paradigms are subject to large variation in XLT performance, especially if the target language is typologically distant to the source (Keung et al., 2020; Zhao et al., 2021; Schmidt et al., 2022). The protocols for model selection in previous XLT work vary broadly, which exacerbates the com- parison of reported XLT results. Some studies (i) do not sufficiently discuss their protocol (Conneau et al., 2020; Xu et al., 2022), while others (ii) tune hyperparameters on the English development splits (Hu et al., 2020; Wu and Dredze, 2020b), or even (iii) perform model selection on the target- language validation sets (Luo et al., 2021; Fang et al., 2021; Zhao …

Content truncated to 500 words