Multi-Granularity Structural Knowledge Distillation for Language Model Compression

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1001 - 1011 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multi-Granularity Structural Knowledge Distillation for Language Model Compression Chang Liu1,2, Chongyang Tao3∗, Jiazhan Feng1, Dongyan Zhao1,2,4,5∗ 1Wangxuan Institute of Computer Technology, Peking University 2Center for Data Science, Peking University 3Microsoft Corporation 4Artificial Intelligence Institute of Peking University 5State Key Laboratory of Media Convergence Production Technology and Systems {liuchang97,fengjiazhan,zhaody}@pku.edu.cn, chotao@microsoft.com Abstract Transferring the knowledge to a small model through distillation has raised great interest in recent years. Prevailing methods transfer the knowledge derived from mono-granularity lan- guage units (e.g., token-level or sample-level), which is not enough to represent the rich seman- tics of a text and may lose some vital knowl- edge. Besides, these methods form the knowl- edge as individual representations or their sim- ple dependencies, neglecting abundant struc- tural relations among intermediate representa- tions. To overcome the problems, we present a novel knowledge distillation framework that gathers intermediate representations from mul- tiple semantic granularities (e.g., tokens, spans and samples) and forms the knowledge as more sophisticated structural relations specified as the pair-wise interactions and the triplet-wise geometric angles based on multi-granularity representations. Moreover, we propose distill- ing the well-organized multi-granularity struc- tural knowledge to the student hierarchically across layers. Experimental results on GLUE benchmark demonstrate that our method out- performs advanced distillation methods. 1 Introduction Recent years have witnessed a surge of pre-trained language models (Devlin et al., 2019; Lewis et al., 2020; Clark et al., 2020; Brown et al., 2020). Build- ing upon the transformer architecture (Vaswani et al., 2017) and pre-trained on large-scale cor- pora using self-supervised objectives, these PLMs have achieved remarkable success in a wide range of natural language understanding and generation tasks. Despite their high performance, these PLMs usually suffer from high computation and memory costs, which hinders them from being deployed ∗Corresponding authors: Chongyang Tao and Dongyan Zhao. into resource-scarce scenarios, e.g., mobile phones and embedded devices. Various attempts have been made to compress the huge PLMs into small ones with minimum performance degradation. As one of the main approaches, knowledge distillation (Hinton et al., 2015) utilizes a large and powerful teacher model to transfer the knowledge to a small student model. Based on the teacher-student framework, Jiao et al. (2020); Wang et al. (2020) distilled the token-level representations and attention dependencies to the student, Sanh et al. (2019); Sun et al. (2019) taught the student to mimic the output logits of the teacher, Sun et al. (2020) enforced the student’s represen- tation to be closed to the teacher’s while pushing negative samples to be far apart. Although proved effective, existing approaches have some flaws. For one thing, these distillation methods only adopted the representations of mono-granularity language units (i.e., token-level or sample-level), while ne- glecting other granularity. For another, their distil- lation objectives either matched the corresponding representations between the teacher and the stu- dent or aligned the attention dependencies, failing to capture more …

Content truncated to 500 words