f-Divergence Minimization for Sequence-Level Knowledge Distillation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10817–10834 July 9-14, 2023 ©2023 Association for Computational Linguistics f-Divergence Minimization for Sequence-Level Knowledge Distillation Yuqiao Wen1,∗, Zichao Li2,∗, Wenyu Du3, Lili Mou1,4 1Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta 2Mila, McGill University 4Canada CIFAR AI Chair, Amii3The University of Hong Kong ∗Equal contribution yq.when@gmail.com, zichao.li@mila.quebec wenyudu@yahoo.com, doublepower.mou@gmail.com Abstract Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one. It has gained increasing attention in the natural language processing community, driven by the demands of compressing ever- growing language models. In this work, we propose an f-DISTILL framework, which for- mulates sequence-level knowledge distillation as minimizing a generalized f-divergence func- tion. We propose four distilling variants under our framework and show that existing SeqKD and ENGINE approaches are approximations of our f-DISTILL methods. We further derive step-wise decomposition for our f-DISTILL, re- ducing intractable sequence-level divergence to word-level losses that can be computed in a tractable manner. Experiments across four datasets show that our methods outperform ex- isting KD approaches, and that our symmetric distilling losses can better force the student to learn from the teacher distribution.1 1 Introduction Increasingly large language models have contin- ued to achieve state-of-the-art performance across various natural language generation tasks, such as data-to-text generation (Lebret et al., 2016; Li and Liang, 2021), summarization (Paulus et al., 2018; Zhang et al., 2020a), and dialogue generation (Li et al., 2016b; Zhang et al., 2020b). However, super- large language models are inaccessible to most users and researchers due to their prohibitively large model size, emphasizing the importance of high-performing, parameter-efficient small neural models. A widely used approach to training small mod- els is knowledge distillation (KD, Hinton et al., 2015), where the small model (known as the stu- dent) learns the knowledge from a much larger model (known as the teacher). KD has shown great 1Our code is available at https://github.com/ MANGA-UOFA/fdistill success in helping smaller models achieve compet- itive performance across a wide range of applica- tions (Sun et al., 2019; Jiao et al., 2020; Shleifer and Rush, 2020). Existing KD approaches can be categorized into two main branches: representation matching and distribution matching. The former aims to imitate the teacher’s real-valued intermediate-layer rep- resentations, say, with mean squared error (Sun et al., 2019; Jiao et al., 2020). Our work focuses on the latter, distribution matching, where the student model learns the teacher’s predictive distribution. Hinton et al. (2015) minimize the cross-entropy loss against the teacher-predicted soft labels, which is equivalent to minimizing the Kullback–Leibler (KL) divergence between the teacher and student. Kim and Rush (2016) propose SeqKD, arguing that KL divergence should be minimized at the se- quence level for language models. However, such an approach tends to learn an overly smooth stu- dent distribution to cover the entire support of the teacher distribution due to the asymmetric nature of the KL divergence. This is often …

Content truncated to 500 words