Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3026–3036 August 1–6, 2021. ©2021 Association for Computational Linguistics 3026 Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains Haojie Pan1,2∗, Chengyu Wang2∗, Minghui Qiu2†, Yichang Zhang2, Yaliang Li2, Jun Huang2 1 Zhejiang Lab 2 Alibaba Group {haojie.phj, chengyu.wcy, minghui.qmh}@alibaba-inc.com {yichang.zyc, yaliang.li, huangjun.hj}@alibaba-inc.com Abstract Pre-trained language models have been ap- plied to various NLP tasks with considerable performance gains. However, the large model sizes, together with the long inference time, limit the deployment of such models in real- time applications. One line of model com- pression approaches considers knowledge dis- tillation to distill large teacher models into small student models. Most of these stud- ies focus on single-domain only, which ig- nores the transferable knowledge from other domains. We notice that training a teacher with transferable knowledge digested across domains can achieve better generalization ca- pability to help knowledge distillation. Hence we propose a Meta-Knowledge Distillation (Meta-KD) framework to build a meta-teacher model that captures transferable knowledge across domains and passes such knowledge to students. Specifically, we explicitly force the meta-teacher to capture transferable knowl- edge at both instance-level and feature-level from multiple domains, and then propose a meta-distillation algorithm to learn single- domain student models with guidance from the meta-teacher. Experiments on public multi- domain NLP tasks show the effectiveness and superiority of the proposed Meta-KD frame- work. Further, we also demonstrate the capa- bility of Meta-KD in the settings where the training data is scarce. 1 Introduction Pre-trained Language Models (PLM) such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) have achieved significant success with the two-stage “pre-training and fine-tuning” process. Despite the performance gain achieved in various NLP tasks, the large number of model parameters ∗H. Pan and C. Wang contributed equally to this work. †M. Qiu is the corresponding author. (a) Learning from an in-domain teacher. (b) Learning from multiple teachers of varied domains. (c) Learning from meta-teacher with multi-domain knowledge. Physics Teacher Physics Teacher Physics Equation Physics Student Math Teacher Physics Equation All-purpose Teacher Physics Equation Physics Student Physics Student Figure 1: A motivation example of academic learning. A physics student may learn physics equations better with a powerful all-purpose teacher. and the long inference time have become the bot- tleneck for PLMs to be deployed in real-time ap- plications, especially on mobile devices (Jiao et al., 2019; Sun et al., 2020; Iandola et al., 2020). Thus, there are increasing needs for PLMs to reduce the model size and the computational overhead while keeping the prediction accuracy. Knowledge Distillation (KD) (Hinton et al., 2015) is one of the promising ways to distill the knowledge from a large “teacher” model to a small “student” model. Recent studies show that KD can be applied to compress PLMs with acceptable per- formance loss (Sanh et al., 2019; Sun et al., 2019b; Jiao et al., 2019; Turc et al., …

Content truncated to 500 words