Adversarial Self-Supervised Data-Free Distillation for Text Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6182–6192, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6182 Adversarial Self-Supervised Data-Free Distillation for Text Classification Xinyin Ma, Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, Weiming Lu∗ College of Computer Science, Zhejiang University {maxinyin, syl, fgf, chen double, chjia, luwm}@zju.edu.cn Abstract Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computation- ally expensive model to a resource-efficient lightweight model. However, most KD algo- rithms, especially in NLP, rely on the acces- sibility of the original training dataset, which may be unavailable due to privacy issues. To tackle this problem, we propose a novel two- stage data-free distillation method, named Ad- versarial self-Supervised Data-Free Distilla- tion (AS-DFD), which is designed for com- pressing large-scale transformer-based models (e.g., BERT). To avoid text generation in dis- crete space, we introduce a Plug & Play Em- bedding Guessing method to craft pseudo em- beddings from the teacher’s hidden knowledge. Meanwhile, with a self-supervised module to quantify the student’s ability, we adapt the dif- ficulty of pseudo embeddings in an adversarial training manner. To the best of our knowledge, our framework is the first data-free distillation framework designed for NLP tasks. We verify the effectiveness of our method on several text classification datasets. 1 Introduction Recently, pre-trained language models (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Raf- fel et al., 2019) have achieved tremendous progress and reached the state-of-the-art performance in var- ious downstream tasks such as text classification (Maas et al., 2011), language inference (Bowman et al., 2015) and question answering (Rajpurkar et al., 2016). These models become an indispens- able part of current models for their transferability and generalizability. ∗* Corresponding author However, such language models are huge in volume and demand highly in computational re- sources, making it impractical in deploying them on portable systems with limited resources (e.g., mobile phones, edge devices) without appropri- ate compression. Recent researches (McCarley, 2019; Gordon et al., 2020; Michel et al., 2019) focus on compressing the large-scale models to a shallow and resource-efficient network via weight pruning (Guo et al., 2019), knowledge distillation (Mukherjee and Awadallah, 2019), weight quan- tization (Zafrir et al., 2019) and parameter shar- ing (Lan et al., 2020). Among them, some meth- ods (Sanh et al., 2019; Sun et al., 2019) draw on the idea of transfer learning, utilizing knowledge distillation (Hinton et al., 2015) to transfer latent representation information embedded in teachers to students. These knowledge distillation meth- ods share some commonalities: they rely on the training data to achieve high accuracy. It will be intractable if we need to compress a model without publicly accessible data. Reasons for that include privacy protection, company assets, safety/security concerns and transmission. Representative samples include GPT2 (Radford et al., 2019), which has not released its training data with fears of abuse …

Content truncated to 500 words