Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13234–13248 July 9-14, 2023 ©2023 Association for Computational Linguistics Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models Qinhong Zhou1,3, Zonghan Yang1,3, Peng Li2,4,†, Yang Liu1,2,3,4,† 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 2Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China 3Beijing National Research Center for Information Science and Technology 4Shanghai Artificial Intelligence Laboratory, Shanghai, China Abstract Conventional knowledge distillation (KD) methods require access to the internal infor- mation of teachers, e.g., logits. However, such information may not always be accessible for large pre-trained language models (PLMs). In this work, we focus on decision-based KD for PLMs, where only teacher decisions (i.e., top-1 labels) are accessible. Considering the infor- mation gap between logits and decisions, we propose a novel method to estimate logits from the decision distributions. Specifically, deci- sion distributions can be both derived as a func- tion of logits theoretically and estimated with test-time data augmentation empirically. By combining the theoretical and empirical estima- tions of the decision distributions together, the estimation of logits can be successfully reduced to a simple root-finding problem. Extensive ex- periments show that our method significantly outperforms strong baselines on both natural language understanding and machine reading comprehension datasets.1 1 Introduction Various natural language processing (NLP) tasks have witnessed promising performance from large pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020). However, PLMs are usually computa- tionally expensive and memory intensive, hinder- ing their deployment on resource-limited devices. Knowledge distillation (KD) (Hinton et al., 2015) is a popular technique to transfer knowledge from large PLMs to lightweight models. Previous KD works utilize various types of internal information from the teacher model, such as output logits (Sanh et al., 2019; Tang et al., 2019; Liu et al., 2020), hidden states (Sun et al., 2019b; Jiao et al., 2020), † Peng Li (lipeng@air.tsinghua.edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn) are corresponding authors. 1Our code is available at https://github.com/THUNLP-MT/ DBKD-PLM and attention maps (Li et al., 2020). In real-world applications, however, these types of information are sometimes not accessible due to commercial and privacy issues (Brown et al., 2020; Ouyang et al., 2022). Specifically, large-scale PLMs usu- ally only provide decisions (i.e., top-1 labels) to users. Motivated by this scenario, we investigate the task of decision-based KD (Wang, 2021) for PLMs, in which only decisions of teacher predic- tions are available. The information gap between teacher decisions and its internal states is the major challenge for the task. A straightforward approach for decision- based KD is to treat teacher decisions as ground truth labels and use these labels to train a student model (Zhang et al., 2022; Sanyal et al., 2022). However, previous work reveals that logits contain rich knowledge (Hinton et al., 2015), relying only on decisions obviously suffers from information loss. To alleviate …

Content truncated to 500 words