Stop Measuring Calibration When Humans Disagree

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892 - 1915 December 7-11, 2022 ©2022 Association for Computational Linguistics Stop Measuring Calibration When Humans Disagree Joris Baan1, Wilker Aziz1, Barbara Plank2,3,4, Raquel Fernández1 1University of Amsterdam, 2IT University of Copenhagen, 3MCML Munich, 4LMU Munich {j.s.baan,w.aziz,raquel.fernandez}@uva.nl, b.plank@lmu.de Abstract Calibration is a popular framework to evalu- ate whether a classifier knows when it does not know—i.e., its predictive probabilities are a good indication of how likely a prediction is to be correct. Correctness is commonly estimated against the human majority class. Recently, calibration to human majority has been mea- sured on tasks where humans inherently dis- agree about which class applies. We show that measuring calibration to human majority given inherent disagreements is theoretically prob- lematic, demonstrate this empirically on the ChaosNLI dataset, and derive several instance- level measures of calibration that capture key statistical properties of human judgements— class frequency, ranking and entropy.1 1 Introduction Neural text classifiers are becoming more powerful but increasingly difficult to interpret (Rogers et al., 2020). In response, the demand for transparency and trust in their predictions is growing (Yin et al., 2019; Bansal et al., 2019; Bianchi and Hovy, 2021). One step towards understanding when to trust pre- dictions is to evaluate whether models know when they do not know—i.e., whether predictive proba- bilities are a good indication of how likely a predic- tion is to be correct—known as calibration. This is crucial in user-facing and high-stake applications. Calibration, and particularly the Expected Cali- bration Error (ECE; Naeini et al., 2015; Guo et al., 2017), is widely studied in Machine Learning and Computer Vision (Mena et al., 2021), and is gaining increased attention in Natural Language Process- ing (NLP; Desai and Durrett, 2020; Kong et al., 2020; Jiang et al., 2021; Dan and Roth, 2021). An important implicit assumption in the widely used definition of perfect calibration proposed by Guo et al. (2017) is that predictions are either right 1Code available at https://github.com/jsbaan/ calibration-on-disagreement-data. or wrong—in other words, that the true class dis- tribution, i.e., human judgement distribution, is deterministic (one-hot). However, for many prob- lems, while categories exist, their boundaries are fluid: there exists inherent disagreement about labels. This means that gold labels are at best an idealization—as irreconcilable disagreement is abundant (Plank et al., 2014; Aroyo and Welty, 2015; Jamison and Gurevych, 2015; Palomaki et al., 2018; Pavlick and Kwiatkowski, 2019). Evidence for this can be found in various tasks, including those which involve linguistic and subjective judge- ments (Akhtar et al., 2020; Basile et al., 2021). Surprisingly, however, while limitations of calibra- tion are studied (§2), this fundamental assumption is ignored. In this work, we show that popular calibration metrics—such as ECE—are not applicable to data with inherent human disagreement (§3). We pro- pose an alternative, instance-level notion of cal- ibration based on human uncertainty, and opera- tionalize it with several measures that capture key statistics of the human judgement distribution other than matching the majority vote …

Content truncated to 500 words