Model Selection for Cross-lingual Transfer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5675–5687 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5675 Model Selection for Cross-Lingual Transfer Yang Chen and Alan Ritter School of Interactive Computing Georgia Institute of Technology {yang.chen, alan.ritter}@cc.gatech.edu Abstract Transformers that are pre-trained on multilin- gual corpora, such as, mBERT and XLM- RoBERTa, have achieved impressive cross- lingual transfer capabilities. In the zero-shot transfer setting, only English training data is used, and the fine-tuned model is evaluated on another target language. While this works sur- prisingly well, substantial variance has been observed in target language performance be- tween different fine-tuning runs, and in the zero-shot setup, no target-language develop- ment data is available to select among multi- ple fine-tuned models. Prior work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, of- ten resulting in suboptimal choices. In this pa- per, we show that it is possible to select con- sistently better models when small amounts of annotated data are available in auxiliary pivot languages. We propose a machine learning ap- proach to model selection that uses the fine- tuned model’s own internal representations to predict its cross-lingual capabilities. In ex- tensive experiments we find that this method consistently selects better models than English validation data across twenty five languages (including eight low-resource languages), and often achieves results that are comparable to model selection using target language develop- ment data.1 1 Introduction Pre-trained Transformers (Vaswani et al., 2017; De- vlin et al., 2019) have achieved state-of-the-art re- sults on a range of NLP tasks, often approaching human inter-rater agreement (Joshi et al., 2020a). These models have also been demonstrated to learn effective cross-lingual representations, even with- out access to parallel text or bilingual lexicons (Wu and Dredze, 2019; Pires et al., 2019). 1Our code and data is available at: https://github. com/edchengg/model_selection In the zero-shot transfer learning, training and de- velopment data are only assumed in a high resource source language (e.g. English), and performance is evaluated on another target language. Because no target language annotations are assumed, source language data is typically used to select among models that are fine-tuned with different hyper- parameters and random seeds. However, recent work has shown that English dev accuracy does not always correlate well with target language perfor- mance (Keung et al., 2020). In this paper, we propose an alternative strategy for model selection in zero-shot transfer. Our ap- proach, dubbed Learned Model Selection (LMS), learns a function that scores the compatibility be- tween a fine-tuned multilingual Transformer, and a target language. The compatibility score is calculated based on features of the multilingual model’s learned representations. This is done by aggregating representations over an unlabeled target language text corpus after fine-tuning on source language data. We show that these model- specific features effectively capture information about how the cross-lingual representations will transfer. We also make use of language embed- dings from the lang2vec package …

Content truncated to 500 words