AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8449–8465 July 9-14, 2023 ©2023 Association for Computational Linguistics AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression Siyue Wu1, Hongzhan Chen1, Xiaojun Quan1∗, Qifan Wang2 and Rui Wang3 1School of Computer Science and Engineering, Sun Yat-sen University, China 2Meta AI 3Vipshop (China) Co., Ltd., China 1 {wusy39, chenhzh59}@mail2.sysu.edu.cn, quanxj3@mail.sysu.edu.cn 2wqfcr@fb.com 3mars198356@hotmail.com Abstract Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowl- edge distillation methods suffer from two lim- itations. First, the student model simply imi- tates the teacher’s behavior while ignoring the underlying reasoning. Second, these methods usually focus on the transfer of sophisticated model-specific knowledge but overlook data- specific knowledge. In this paper, we present a novel attribution-driven knowledge distilla- tion approach, which explores the token-level rationale behind the teacher model based on Integrated Gradients (IG) and transfers attri- bution knowledge to the student model. To enhance the knowledge transfer of model rea- soning and generalization, we further explore multi-view attribution distillation on all poten- tial decisions of the teacher. Comprehensive experiments are conducted with BERT on the GLUE benchmark. The experimental results demonstrate the superior performance of our approach to several state-of-the-art methods. 1 Introduction Transformer-based pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have aroused widespread interest among Natural Language Pro- cessing (NLP) researchers in recent years. These language models are first pre-trained on large-scale unlabeled corpora to learn the general representa- tion of language, and then fine-tuned on specific downstream tasks to effectively transfer the gen- eral knowledge to target domains. This pre-training and fine-tuning paradigm leads to state-of-the-art performances in various NLP tasks such as nat- ural language understanding. However, with the rapid growth of the model scale, the deployment of large-scale PLMs becomes challenging, especially in low-resource scenarios. To this end, a variety ∗Corresponding author. Input Layer Output Layer Intermediate Layer Entailment Response-based Knowledge Feature-based Knowledge Relation-based Knowledge Attribution-based Knowledge Attribution Score 0.0 0.4 0.2 ❓ Question: What land was ceded to Spain ? 📜 Sentence: It ceded French Louisiana west of the Mississippi River ( including New Orleans ) to its ally Spain , in compensation for Spain ' s loss to Britain of Florida … } features from other data Figure 1: An example from the QNLI dataset (Rajpurkar et al., 2016) to illustrate different knowledge distillation techniques including the proposed attribution-driven method. Darker colors mean larger attribution scores. of model compression techniques have been devel- oped. Among them, knowledge distillation (KD) (Hinton et al., 2015) is a newly emerging technol- ogy that aims to obtain a small student model by distilling knowledge from a large teacher model and achieve comparable performance. Existing knowledge distillation methods can be divided into three categories, namely response- based, feature-based, and relation-based (Gou et al., 2021). While response-based methods (Turc et al., 2019) directly distill the final output, e.g. prob- ability …

Content truncated to 500 words