HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3126–3136 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3126 HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression Chenhe Dong1, Yaliang Li2, Ying Shen1∗, Minghui Qiu2∗ 1 Sun Yat-sen University 2 Alibaba Group dongchh@mail2.sysu.edu.cn, sheny76@mail.sysu.edu.cn {yaliang.li, minghui.qmh}@alibaba-inc.com Abstract On many natural language processing tasks, large pre-trained language models (PLMs) have shown overwhelming performances com- pared with traditional neural network meth- ods. Nevertheless, their huge model size and low inference speed have hindered the deploy- ment on resource-limited devices in practice. In this paper, we target to compress PLMs with knowledge distillation, and propose a hierarchical relational knowledge distillation (HRKD) method to capture both hierarchical and domain relational information. Specif- ically, to enhance the model capability and transferability, we leverage the idea of meta- learning and set up domain-relational graphs to capture the relational information across dif- ferent domains. And to dynamically select the most representative prototypes for each domain, we propose a hierarchical compare- aggregate mechanism to capture hierarchical relationships. Extensive experiments on pub- lic multi-domain datasets demonstrate the su- perior performance of our HRKD method as well as its strong few-shot learning ability. For reproducibility, we release the code at https: //github.com/cheneydon/hrkd. 1 Introduction Large pre-trained language models (PLMs) (e.g., BERT (Devlin et al., 2019)) have demonstrated their outperforming performances on a wide range of NLP tasks, such as machine translation (CON- NEAU and Lample, 2019; Zhu et al., 2020), sum- marization (Zhang et al., 2019; Liu and Lapata, 2019), and dialogue generation (Bao et al., 2020; Zheng et al., 2020). However, their large size and slow inference speed have hindered practi- cal deployments, such as deploying on resource- constrained devices. To solve the above problem, many compression techniques for PLMs have been proposed, such as ∗Corresponding author. quantization (Shen et al., 2020), weight pruning (Michel et al., 2019), and knowledge distillation (KD) (Sun et al., 2019; Jiao et al., 2020). Due to the plug-and-play feasibility of KD, it is the most commonly used method in practice, and we focus on it in this work. The purpose of KD is to trans- fer knowledge from a larger teacher model to a smaller student model (Hinton et al., 2015). Tra- ditional KD methods only leverage single-domain knowledge, i.e., transferring the knowledge of the teacher model to the student model domain by do- main. However, as stated in the purpose of trans- fer learning, the model performance on target do- mains can be improved by transferring the knowl- edge from different but related source domains (Lu et al., 2015), thus the cross-domain knowledge also plays an important role. In addition, several recent works have also proved the advantage of cross- domain knowledge, and many multi-domain KD methods have been proposed. For example, Peng et al. (2020); Yang et al. (2020) demonstrate the effectiveness of distilling knowledge from multiple teachers in different domains; Liu et al. (2019a,b) show that jointly distilling the student models of different domains …

Content truncated to 500 words