Bag of Tricks for In-Distribution Calibration of Pretrained Transformers

Findings of the Association for Computational Linguistics: EACL 2023, pages 551–563 May 2-6, 2023 ©2023 Association for Computational Linguistics Bag of Tricks for In-Distribution Calibration of Pretrained Transformers Jaeyoung Kim VUNO, Inc. jaeyoung.kim@vuno.co Dongbin Na VUNO, Inc. dongbin.na@vuno.co Sungchul Choi Pukyong National University sc82.choi@pknu.ac.kr Sungbin Lim∗ Korea University sungbin@korea.ac.kr Abstract While pre-trained language models (PLMs) have become a de-facto standard promoting the accuracy of text classification tasks, re- cent studies (Kong et al., 2020; Dan and Roth, 2021) find that PLMs often predict over-confidently. Although various calibration methods have been proposed, such as ensem- ble learning and data augmentation, most of the methods have been verified in computer vision benchmarks rather than in PLM-based text classification tasks. In this paper, we present an empirical study on confidence cali- bration for PLMs, addressing three categories, including confidence penalty losses, data aug- mentations, and ensemble methods. We find that the ensemble model overfitted to the train- ing set shows sub-par calibration performance and also observe that PLMs trained with con- fidence penalty loss have a trade-off between calibration and accuracy. Building on these observations, we propose the Calibrated PLM (CALL), a combination of calibration tech- niques. The CALL complements the draw- backs that may occur when utilizing a cali- bration method individually and boosts both classification and calibration accuracy. Design choices in CALL’s training procedures are ex- tensively studied, and we provide a detailed analysis of how calibration techniques affect the calibration performance of PLMs. 1 Introduction Trustworthy deployment of machine learning appli- cations requires accurate and calibrated predictions to instill their reliability and help users be less con- fused about models’ decisions (Xiao and Wang, 2019; Liu et al., 2020). However, modern deep neural networks (DNNs) produce miscalibrated predictions, i.e., a mismatch between a model’s confidence and its correctness. One of the reasons is that an over-parameterized ∗Corresponding author. This work is partially done at UNIST. (a) BERT (b) RoBERTa Figure 1: Reliability diagrams (DeGroot and Fienberg, 1983) on TREC (Li and Roth, 2002) with PLMs. A dashed line implies a perfect calibration while PLMs generally show over-confident predictions. classifier typically produces over-confident predic- tions (Guo et al., 2017). Moreover, the miscali- bration can be exacerbated when DNNs make pre- dictions on test data different from the training distribution, i.e., distribution shift (Ovadia et al., 2019). To obtain the well-calibrated predictions, many pioneering studies have shown the calibration effect of ensemble and regularization techniques focused on computer vision benchmarks. Ensemble learn- ing has become one of the standard approaches to reduce calibration errors (Lakshminarayanan et al., 2017; Bonab and Can, 2019). Pereyra et al. (2017) propose the entropy regularized loss which penal- izes confident output distributions in order to re- duce overfitting. Hongyi Zhang (2018); Hendrycks et al. (2020) demonstrate that DNNs trained on diverse augmented data are less prone to produce over-confident predictions, leading to the calibra- tion benefit under the distribution shift. Intense research effort has focused on improving the calibration performance of vision models on image datasets. However, …

Content truncated to 500 words