Posterior Calibrated Training on Sentence Classification Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2723–2730 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2723 Posterior Calibrated Training on Sentence Classification Tasks Taehee Jung1 Dongyeop Kang2 Hua Cheng3 Lucas Mentch1 Thomas Schaaf 3 1Department of Statistics, University of Pittsburgh, Pittsburgh, PA, USA 2School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA 33M|M*Modal, Pittsburgh, PA, USA {taj41,lkm31}@pitt.edu dongyeok@cs.cmu.edu {hcheng,tschaaf}@mmm.com Abstract Most classification models work by first pre- dicting a posterior probability distribution over all classes and then selecting that class with the largest estimated probability. In many set- tings however, the quality of posterior proba- bility itself (e.g., 65% chance having diabetes), gives more reliable information than the final predicted class alone. When these methods are shown to be poorly calibrated, most fixes to date have relied on posterior calibration, which rescales the predicted probabilities but often has little impact on final classifications. Here we propose an end-to-end training procedure called posterior calibrated (PosCal) training that directly optimizes the objective while min- imizing the difference between the predicted and empirical posterior probabilities. We show that PosCal not only helps reduce the calibra- tion error but also improve task performance by penalizing drops in performance of both objectives. Our PosCal achieves about 2.5% of task performance gain and 16.1% of cali- bration error reduction on GLUE (Wang et al., 2018) compared to the baseline. We achieved the comparable task performance with 13.2% calibration error reduction on xSLUE (Kang and Hovy, 2019), but not outperforming the two-stage calibration baseline. PosCal training can be easily extendable to any types of classi- fication tasks as a form of regularization term. Also, PosCal has the advantage that it incre- mentally tracks needed statistics for the cali- bration objective during the training process, making efficient use of large training sets1. 1 Introduction Classification systems, from simple logistic regres- sion to complex neural network, typically predict posterior probabilities over classes and decide the final class with the maximum probability. The 1Code is publicly available at https://github.com/ THEEJUNG/PosCal/ model’s performance is then evaluated by how ac- curate the predicted classes are with respect to out- of-sample, ground-truth labels. In some cases, how- ever, the quality of posterior estimates themselves must be carefully considered as such estimates are often interpreted as a measure of confidence in the final prediction. For instance, a well-predicted pos- terior can help assess the fairness of a recidivism prediction instrument (Chouldechova, 2017) or se- lect the optimal number of labels in a diagnosis code prediction (Kavuluru et al., 2015). Guo et al. (2017) showed that a model with high classification accuracy does not guarantee good posterior estimation quality. In order to correct the poorly calibrated posterior probability, existing calibration methods (Zadrozny and Elkan, 2001; Platt et al., 1999; Guo et al., 2017; Kumar et al., 2019) generally rescale the posterior distribution predicted from the classifier after training. Such post-processing calibration methods re-learn an ap- propriate distribution from a held-out validation set and then apply …

Content truncated to 500 words