Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers

Findings of the Association for Computational Linguistics: ACL 2022, pages 3673 - 3684 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers Rishabh Singh and Shirin Goshtasbpour Department of Computer Science ETH Zurich, Switzerland Abstract Modern NLP classifiers are known to return un- calibrated estimations of class posteriors. Ex- isting methods for posterior calibration rescale the predicted probabilities but often have an adverse impact on final classification accuracy, thus leading to poorer generalization. We pro- pose an end-to-end trained calibrator, Platt- Binning, that directly optimizes the objective while minimizing the difference between the predicted and empirical posterior probabilities. Our method leverages the sample efficiency of Platt scaling and the verification guarantees of histogram binning, thus not only reducing the calibration error but also improving task perfor- mance. In contrast to existing calibrators, we perform this efficient calibration during train- ing. Empirical evaluation of benchmark NLP classification tasks echoes the efficacy of our proposal. 1 Introduction Deep learning has proven to be tremendously at- tractive for researchers in fields such as physics, biology, and manufacturing, to name a few (Baldi et al., 2014; Anjos et al., 2015; Bergmann et al., 2014). However, these are fields in which rep- resenting model uncertainty is of crucial impor- tance (Gal and Ghahramani, 2016). A common way to incorporate DNNs in other fields is to use the predictions of a trained classifier for decision making in a downstream task. In some cases the effectiveness of the decisions depends on a util- ity function and it is not enough to simply predict the most likely label for each example. What is needed instead is to quantify model uncertainty about the predictions. Despite promising perfor- mance in supervised learning benchmarks in terms of accuracy, DNNs are poor at quantifying predic- tive uncertainty, and tend to produce overconfident predictions. Overconfident incorrect predictions can be harmful or offensive in NLP applications (Amodei et al., 2016), hence proper uncertainty quantification is crucial in practice. Probabilistic uncertainty in machine learning translates to esti- mation of the probability mass function p(y|x) by the model, where x is the input sample and y is a class label. Recent works have shown that state- of-the art structured prediction models are poorly calibrated. Therefore, blindly using the output of the softmax function output as the model uncer- tainty is misleading (Kumar and Sarawagi, 2019; Dong et al., 2018; Nguyen and O’Connor, 2015). We are interested in calibrating the posterior es- timates, i.e. we wish to get posterior probability estimations that reflect the true probability of the classes. The probability that a system outputs for an event should reflect the true frequency of that event: if an automated diagnosis system says 1,000 patients have cancer with probability 0.1, approx- imately 100 of them should indeed have cancer (Kumar et al., 2019). Even if the actual mechanism might be difficult to interpret, a calibrated model at least gives us a signal that it “knows what it doesn’t know,” thereby making …

Content truncated to 500 words