Calibrating Zero-shot Cross-lingual (Un-)structured Predictions

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2648 - 2674 December 7-11, 2022 ©2022 Association for Computational Linguistics Calibrating Zero-shot Cross-lingual (Un-)structured Predictions Zhengping Jiang, Anqi Liu, Benjamin Van Durme Johns Hopkins University {zjiang31, aliu74, vandurme}@jhu.edu Abstract We investigate model calibration in the setting of zero-shot cross-lingual transfer with large- scale pre-trained language models. The level of model calibration is an important metric for evaluating the trustworthiness of predic- tive models. There exists an essential need for model calibration when natural language models are deployed in critical tasks. We study different post-training calibration methods in structured and unstructured prediction tasks. We find that models trained with data from the source language become less calibrated when applied to the target language and that calibra- tion errors increase with intrinsic task difficulty and relative sparsity of training data. Moreover, we observe a potential connection between the level of calibration error and an earlier pro- posed measure of the distance from English to other languages. Finally, our comparison demonstrates that among other methods Tem- perature Scaling (TS) generalizes well to dis- tant languages, but TS fails to calibrate more complex confidence estimation in structured predictions compared to more expressive alter- natives like Gaussian Process Calibration. 1 Introduction While deep neural networks, especially large pre- trained language models, have driven striking im- provements on various standard benchmarks (Wang et al., 2018, 2019), it is never a good practice to assume their predictions are accurate and should be taken blindly. In many cases, it is important to un- derstand “what a model does not know” through its estimation of its uncertainty. For example, reliable model confidence is important in high-stakes do- mains (Begoli et al., 2019; Zhong et al., 2019), or when downstream tasks leverage confidence scores to mitigate error propagation (Chang et al., 2007). Moreover, accurate confidence can serve as a mea- sure of the value of information in iterative data collection or human-in-the-loop learning (Zhang et al., 2019; Chaudhary et al., 2021). Figure 1: Averaged Expected Calibration Error (ECE) before and after temperature scaling on English (top) and Arabic (bottom) xlm-roberta-large; lower is better. Multiple bars for a task reference full-data, low-data, and very-low-data (from left to right) settings. Models appear less calibrated when transferred to other languages while temperature scaling remains effective. Whether the model confidence is accurate is usu- ally measured by how well it matches the observa- tional data – through confidence calibration (Guo et al., 2017). Yet modern neural networks are criti- cized for being overconfident with their predictions, given their increased capacity to fit the training dataset (Guo et al., 2017). This problem is exac- erbated by domain-shift (Ovadia et al., 2019) or zero-/few-shot transfer (Liu et al., 2018). An im- portant task that is often concerned with such data shift is zero-shot cross-lingual transfer, which has been viewed as a natural extension to domain adap- tation (Ruder et al., 2019; Xian et al., 2021). Existing studies in natural language processing have …

Content truncated to 500 words