Calibrating Student Models for Emotion-related Tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9266 - 9278 December 7-11, 2022 ©2022 Association for Computational Linguistics Calibrating Student Models for Emotion-related Tasks Mahshid Hosseini Cornelia Caragea Computer Science University of Illinois at Chicago mhosse4@uic.edu cornelia@uic.edu Abstract Knowledge Distillation (KD) is an effective method to transfer knowledge from one net- work (a.k.a. teacher) to another (a.k.a. student). In this paper, we study KD on the emotion- related tasks from a new perspective: calibra- tion. We further explore the impact of the mixup data augmentation technique on the dis- tillation objective and propose to use a sim- ple yet effective mixup method informed by training dynamics for calibrating the student models. Underpinned by the regularization im- pact of the mixup process by providing better training signals to the student models using training dynamics, our proposed mixup strat- egy gradually enhances the student model’s calibration while effectively improving its per- formance. We evaluate the calibration of pre- trained language models through knowledge distillation over three tasks of emotion detec- tion, sentiment analysis, and empathy detection. By conducting extensive experiments on differ- ent datasets, with both in-domain and out-of- domain test sets, we demonstrate that student models distilled from teacher models trained using our proposed mixup method obtained the lowest Expected Calibration Errors (ECEs) and best performance on both in-domain and out- of-domain test sets. 1 Introduction It has been shown that transferring knowledge from a teacher model with desired high performance to a student model, through knowledge distillation, can lead to better performance of student models distilled (Furlanello et al., 2018; Yim et al., 2017). However, little is known about the impact of the distillation process on the calibration of the student model. Evaluating the uncertainty of a model’s predictions is crucial, specifically in applications where the cost of an error is high. For instance, in a computer-assisted therapy session, an accurate and calibrated emotion or empathy detection model can inform the doctor when a model’s predictions should (or should not) be trusted, which is help- ful for them in deciding the preferred treatment for patients. In this work, we aim to shed light on the impact of knowledge distillation on the calibra- tion of the student models on emotion-related tasks. Calibration measures the discrepancy between the correctness of the prediction (i.e., accuracy) and the (empirical) probability that a model assigns to a prediction (i.e., confidence). A well-calibrated model knows how often it is correct or wrong; pre- dicting an event with p confidence shall empirically be true p of the time (Guo et al., 2017). Recently, a large body of work has investigated why neural networks have become miscalibrated (Platt et al., 1999; Niculescu-Mizil and Caruana, 2005; Nguyen and O’Connor, 2015a; Kuleshov and Liang, 2015; Kuleshov and Ermon, 2016; Guo et al., 2017; Desai and Durrett, 2020). More re- cent attention, however, has focused on methods to alleviate this problem. Specifically on natural language processing tasks, Guo et al. (2017) pro- posed a simple …

Content truncated to 500 words