CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation

Proceedings of the 29th International Conference on Computational Linguistics, pages 4707–4713 October 12–17, 2022. 4707 CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation Md Akmal Haidar1∗ Mehdi Rezagholizadeh1 Abbas Ghaddar1 Khalil Bibi1 Philippe Langlais2 Pascal Poupart3 1Huawei Noah’s Ark Lab 2 RALI/DIRO, Université de Montréal, Canada 3 David R. Cheriton School of Computer Science, University of Waterloo {mehdi.rezagholizadeh,abbas.ghaddar}@huawei.com felipe@iro.umontreal.ca, ppoupart@uwaterloo.ca Abstract Knowledge distillation (KD) is an efficient framework for compressing large-scale pre- trained language models. Recent years have seen a surge of research aiming to im- prove KD by leveraging Contrastive Learning, Intermediate Layer Distillation, Data Aug- mentation, and Adversarial Training. In this work, we propose a learning based data aug- mentation technique tailored for knowledge distillation, called CILDA. To the best of our knowledge, this is the first time that intermedi- ate layer representations of the main task are used in improving the quality of augmented samples. More precisely, we introduce an aug- mentation technique for KD based on inter- mediate layer matching using contrastive loss to improve masked adversarial data augmenta- tion. CILDA outperforms existing state-of-the- art KD approaches on the GLUE benchmark, as well as in an out-of-domain evaluation. 1 Introduction The exponentially increasing size of pre-trained large language models (Devlin et al., 2019; Liu et al., 2020; Raffel et al., 2020; Brown et al., 2020) has been a persistent concern regarding the effi- ciency and scalability of Natural Language Under- standing (NLU) in real world applications. Knowl- edge Distillation (KD) (Buciluˇa et al., 2006; Hin- ton et al., 2014) is a technique for transferring the knowledge from a large-scale model (called teacher) to a smaller one (called student), so that the latter model can be employed on edge device (Sanh et al., 2019a; Tang et al., 2019; Mukherjee and Awadallah, 2020; Li et al., 2021). This is done by minimizing the KL divergence between the teacher and student probabilistic outputs. ∗ Work done while at Huawei. Numerous techniques have been exploited re- cently to increase the knowledge transfer beyond logits matching. For instance, it has been found beneficial to perform distillation on the internal components (parameters) of the teacher and stu- dent, which is known as Intermediate Layer Dis- tillation (Sun et al., 2019, 2020b; Passban et al., 2021; Wang et al., 2020a,b; Fu et al., 2021; Wu et al., 2021). Data Augmentation has also been successful for KD (Jiao et al., 2019; Shen et al., 2020; Qu et al., 2021), as researchers have found that the student has less opportunity to acquire useful information from the teacher when limited data are available for training (Kamalloo et al., 2021, 2022; Jafari et al., 2021a). Adversarial Training was also employed in KD (Zhu et al., 2019; Rashid et al., 2020, 2021; He et al., 2021) to improve the robustness and gen- eralization, as the student may predict inconsistent outputs with slight distortion to the data distribu- tions (Li et al., 2021). Recently, Contrastive Learn- ing (Gutmann and Hyvärinen, 2010; Hjelm et al., 2018; Arora et al., …

Content truncated to 500 words