On the Calibration of Massively Multilingual Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4310 - 4323 December 7-11, 2022 ©2022 Association for Computational Linguistics On the Calibration of Massively Multilingual Language Models Kabir Ahuja1 Sunayana Sitaram1 Sandipan Dandapat2 Monojit Choudhury2 1 Microsoft Research, India 2 Microsoft R&D, India {t-kabirahuja,sadandap,sunayana.sitaram,monojitc}@microsoft.com Abstract Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in evaluating these models for their performance on a variety of tasks and languages, little at- tention has been paid on how well calibrated these models are with respect to the confidence in their predictions. We first investigate the calibration of MMLMs in the zero-shot set- ting and observe a clear case of miscalibra- tion in low-resource languages or those which are typologically diverse from English. Next, we empirically show that calibration methods like temperature scaling and label smoothing do reasonably well in improving calibration in the zero-shot scenario. We also find that few- shot examples in the language can further help reduce calibration errors, often substantially. Overall, our work contributes towards building more reliable multilingual models by highlight- ing the issue of their miscalibration, understand- ing what language and model-specific factors influence it, and pointing out the strategies to improve the same. 1 Introduction Massively Multilingual Language Models (MMLMs) like mBERT (Devlin et al., 2019), XLMR (Conneau et al., 2020), mT5 (Xue et al., 2021) and mBART (Liu et al., 2020) have been surprisingly effective at zero-shot cross-lingual transfer i.e. when fine-tuned on an NLP task in one language, they often tend to generalize reasonably well in languages unseen during fine-tuning. These models have been evaluated for their per- formance across a range of multilingual tasks (Pan et al., 2017; Nivre et al., 2018; Conneau et al., 2018) and numerous methods like adapters (Pfeiffer et al., 2020), sparse fine-tuning (Ansell et al., 2022) and few-shot learning (Lauscher et al., 2020) have been proposed to further improve performance of cross- lingual transfer. Despite these developments, there has been lit- tle to no attention paid to the calibration of these models across languages i.e. how reliable the con- fidence predictions of these models are. As these models find their way more and more into the real word applications with safety implications, like Hate Speech Detection (Davidson et al., 2017; Deshpande et al., 2022) it becomes important to only take extreme actions for high confidence pre- dictions by the model (Sarkar and KhudaBukhsh, 2021). Hence, calibrated confidences are desirable to have when deploying such systems in practice. Guo et al. (2017) showed that modern neural net- works used for Image Recognition (He et al., 2016) though perform much better than the ones intro- duced decades ago (Lecun et al., 1998), but are sig- nificantly worse calibrated and often over-estimate their confidence on incorrect predictions. For NLP tasks specifically, Desai and Durrett (2020) showed that classifiers trained using pre-trained transformer-based models (Devlin et al., 2019) are well calibrated both in-domain …

Content truncated to 500 words