Towards Zero-Shot Knowledge Distillation for Natural Language Processing

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6551–6561 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6551 Towards Zero-Shot Knowledge Distillation for Natural Language Processing Ahmad Rashid1, Vasileios Lioutas2∗, Abbas Ghaddar1, Mehdi Rezagholizadeh1 1Huawei Noah’s Ark Lab, 2University of British Columbia ahmad.rashid@huawei.com, contact@vlioutas.com, abbas.ghaddar@huawei.com, mehdi.rezagholizadeh@huawei.com Abstract Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learn- ing based natural language processing (NLP) solutions. In its regular manifestations, KD requires access to the teacher’s training data for knowledge transfer to the student network. However, privacy concerns, data regulations and proprietary reasons may prevent access to such data. We present, to the best of our knowl- edge, the first work on Zero-shot Knowledge Distillation for NLP, where the student learns from the much larger teacher without any task specific data. Our solution combines out-of- domain data and adversarial training to learn the teacher’s output distribution. We investi- gate six tasks from the GLUE benchmark and demonstrate that we can achieve between 75% and 92% of the teacher’s classification score (accuracy or F1) while compressing the model 30 times. 1 Introduction Deep learning based natural learning processing (NLP) systems have become state-of-the-art on many applications such as machine translation (MT) (Vaswani et al., 2017; Lioutas and Guo, 2020), natural language understanding (NLU) (De- vlin et al., 2019) and language generation (Brown et al., 2020) among others. These models are in- creasingly trained on huge corpora and with bil- lions of trainable parameters (Brown et al., 2020). This is prohibitive for deploying them on edge de- vices as well as maintaining them on servers. More- over, training and evaluating them leaves a signifi- cant environmental footprint (Strubell et al., 2019) wherein avoiding the resource hungry training is very challenging (Ghaddar and Langlais, 2019) and may be unavoidable (Li et al., 2020). Model com- ∗Work done during an internship at Huawei Noah’s Ark Lab. pression approaches make it feasible to employ current state of the art models on edge devices. Model compression (Sanh et al., 2019; Jiao et al., 2019) has received a lot of attention in the NLP community due to the aforementioned rea- sons. Some of the algorithms include model prun- ing (See et al., 2016), quantization (Shen et al., 2019), low-rank matrix factorization (Sainath et al., 2013; Lioutas et al., 2019) and knowledge distil- lation (KD) (Buciluˇa et al., 2006; Hinton et al., 2015). KD is one of the most commonly used, appli- cation and model agnostic, compression and en- sembling algorithm. It is one of the most widely researched algorithms for compressing transformer based language models (Rogers et al., 2020; Rashid et al., 2021; Passban et al., 2021; Jafari et al., 2021; Wu et al., 2020; Kamalloo et al., 2021). In KD, the student needs to be trained with the teacher’s train- ing data so as to prevent loss of accuracy. However, we can not assume this access for many practical problems. Some of the concerns …

Content truncated to 500 words