Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9793 - 9803 December 7-11, 2022 ©2022 Association for Computational Linguistics Hard Gate Knowledge Distillation - Leverage Calibration for a Robust and Reliable Language Model Dongkyu Lee1,3∗ Zhiliang Tian2† Yingxiu Zhao1 Ka Chun Cheung3 Nevin L. Zhang1 1Department of Computer Science and Engineering, HKUST 2College of Computer, National University of Defense Technology 3NVIDIA AI Technology Center, NVIDIA 1{dleear, yzhaocx, lzhang}@cse.ust.hk 2tianzhilianghit@gmail.com 3chcheung@nvidia.com Abstract In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that holds inter- class relations which send a meaningful su- pervision to a student; hence, much effort has been put to find such knowledge to be distilled. In this paper, we explore a question that has been given little attention: “when to distill such knowledge." The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscal- ibration of a student. This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data. We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level. Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model general- ization, but also significantly lowers model cal- ibration error. 1 Introduction In recent years, the deep learning community has achieved marked performance gains across a va- riety of tasks (Brown et al., 2020; Devlin et al., 2018). In the meantime, some deep learning mod- els have become excessively large, limiting their applicability in some scenarios. To cope with the issue, Hinton et al. (2015) proposed knowledge distillation (KD), in which knowledge of a large network, called a teacher network, is transferred to a relatively small model, called a student model. The benefits of KD have been widely witnessed across multiple domains (Romero et al., 2015; Jiao ∗This work was done while Dongkyu was an intern at NVIDIA †Corresponding author et al., 2020). Recently, it has been observed that KD can be used in both reducing model size and improving model generalization (Tang et al., 2021; Furlanello et al., 2018). Hinton et al. (2015) ar- gue that a distribution, defined by a teacher, holds inter-class relations, commonly referred to as the dark knowledge, and that such distribution brings a meaningful supervision to a student. There- fore, a large body of research in KD has viewed a teacher as a source of knowledge and has focused on finding a meaningful knowledge to be trans- ferred (Romero et al., 2015; Bulò et al., 2016; Park et al., 2019; Yuan et al., 2020; Kim et al., 2021). In this work, we focus on when to distill knowl- edge of a teacher. This is a …

Content truncated to 500 words