MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1062–1071 August 1–6, 2021. ©2021 Association for Computational Linguistics 1062 MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation Ahmad Rashid1∗, Vasileios Lioutas2∗†, Mehdi Rezagholizadeh1 1Huawei Noah’s Ark Lab, 2University of British Columbia ahmad.rashid@huawei.com, contact@vlioutas.com, mehdi.rezagholizadeh@huawei.com Abstract The advent of large pre-trained language mod- els has given rise to rapid progress in the field of Natural Language Processing (NLP). While the performance of these models on standard benchmarks has scaled with size, compres- sion techniques such as knowledge distilla- tion have been key in making them practi- cal. We present MATE-KD, a novel text- based adversarial training algorithm which im- proves the performance of knowledge distilla- tion. MATE-KD first trains a masked language model-based generator to perturb text by max- imizing the divergence between teacher and student logits. Then using knowledge distilla- tion a student is trained on both the original and the perturbed training samples. We evalu- ate our algorithm, using BERT-based models, on the GLUE benchmark and demonstrate that MATE-KD outperforms competitive adversar- ial learning and data augmentation baselines. On the GLUE test set our 6 layer RoBERTa based model outperforms BERTLARGE. 1 Introduction Transformers (Vaswani et al., 2017) and transformer-based Pre-trained Language Models (PLMs) (Devlin et al., 2019) are ubiquitous in applications of NLP. They are highly parallelizable and their performance scales well with an increase in model parameters and data. Increasing model parameters depends on the availability of computa- tional resources and PLMs are typically trained on unlabeled data which is cheaper to obtain. Recently, the trillion parameter mark has been breached for PLMs (Fedus et al., 2021) amid seri- ous environmental concerns (Strubell et al., 2019). However, without a change in our current training ∗Equal Contribution † Work done during an internship at Huawei Noah’s Ark Lab. paradigm , training larger models may be unavoid- able (Li et al., 2020). In order to deploy these models for practical applications such as for vir- tual personal assistants, recommendation systems, e-commerce platforms etc. model compression is necessary. Knowledge Distillation (KD) (Buciluˇa et al., 2006; Hinton et al., 2015) is a simple, yet pow- erful knowledge transfer algorithm which is used for neural model compression (Jiao et al., 2019; Sanh et al., 2019), ensembling (Hinton et al., 2015) and multi-task learning (Clark et al., 2019). In NLP, KD for compression has received renewed interest in the last few years. It is one of the most widely researched algorithms for the compression of transformer-based PLMs (Rogers et al., 2020). One key feature which makes KD attractive is that it only requires access to the teacher’s output or logits and not the weights themselves. Therefore, if a trillion parameter model resides on the cloud, an API level access to the teacher’s output is sufficient for KD. Consequently, the algorithm is architecture agnostic, i.e., it can work for any deep learning model and the student can be a …

Content truncated to 500 words