Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5454 - 5467 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models Kabir Ahuja1 ∗ Shanu Kumar2 ∗ Sandipan Dandapat2 Monojit Choudhury1 1 Microsoft Research, India 2 Microsoft R&D, Hyderabad, India {t-kabirahuja,shankum,sadandap,monojitc}@microsoft.com Abstract Massively Multilingual Transformer based Lan- guage Models have been observed to be sur- prisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which helps us build more accurate predictors for tasks where we have test data in very few languages to measure the ac- tual performance of the model. Our approach also lends us the ability to perform a much more robust feature selection, and identify a common set of features that influence zero-shot performance across a variety of tasks. 1 Introduction Multilingual models like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have been recently shown to be surprisingly effective for zero-shot transfer (Pires et al., 2019) (Wu and Dredze, 2019), where on fine-tuning for a task on one or a few languages, called pivots, they can perform well on languages unseen during training. The zero-shot performance however, is often not uniform across the languages and the multilingual models turn out to be much less effective for low re- source languages (Wu and Dredze, 2020; Lauscher et al., 2020) and the languages that are typologi- cally distant from the pivots (Lauscher et al., 2020). What affects the zero-shot transfer across different languages is a subject of considerable interest and importance (K et al., 2020; Pires et al., 2019; Wu and Dredze, 2019; Lauscher et al., 2020), however there is little conclusive evidence and a few papers even show contradictory findings. Lauscher et al. (2020) recently, showed that it is possible to predict the zero shot performance of ∗Equal contribution mBERT and XLM-R on different languages by for- mulating it as a regression problem, with pretrain- ing data size and typological similarities between the pivot and target languages as the input features, and the performance on downstream task as the pre- diction target. Along similar lines Srinivasan et al. (2021) and Dolicki and Spanakis (2021) explore zero-shot performance prediction with a larger set of features and different regression techniques. However, the efficacy of these solutions are severely limited by the lack of training data, that is, the number of languages for which performance metrics are available for a given task. For instance, for most tasks in the popular XTREME-R (Ruder et al., 2021) benchmark, there are data points for 7-11 languages. This not only makes zero-shot performance prediction a challenging problem, but also a very important one because …

Content truncated to 500 words