The Multilingual Amazon Reviews Corpus

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 498–508, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 498 Contrastive Distillation on Intermediate Representations for Language Model Compression Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, Jingjing Liu Microsoft Dynamics 365 AI Research {siqi.sun,zhe.gan,yu.cheng,yuwfan,shuohang.wang,jingjl}@microsoft.com Abstract Existing language model compression meth- ods mostly use a simple L2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. Al- though widely used, this objective by design assumes that all the dimensions of hidden rep- resentations are independent, failing to cap- ture important structural knowledge in the in- termediate layers of the teacher network. To achieve better distillation efficacy, we propose Contrastive Distillation on Intermediate Repre- sentations (CODIR), a principled knowledge distillation framework where the student is trained to distill knowledge through interme- diate layers of the teacher via a contrastive objective. By learning to distinguish posi- tive sample from a large set of negative sam- ples, CoDIR facilitates the student’s exploita- tion of rich information in teacher’s hidden layers. CoDIR can be readily applied to com- press large-scale language models in both pre- training and finetuning stages, and achieves superb performance on the GLUE bench- mark, outperforming state-of-the-art compres- sion methods.1 1 Introduction Large-scale pre-trained language models (LMs), such as BERT (Devlin et al., 2018), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), have brought revolutionary advancement to the NLP field (Wang et al., 2018). However, as new- generation LMs grow more and more into behe- moth size, it becomes increasingly challenging to deploy them in resource-deprived environment. Naturally, there has been a surge of research inter- est in developing model compression methods (Sun et al., 2019; Sanh et al., 2019; Shen et al., 2019) 1Code will be released at https://github.com/ intersun/CoDIR. to reduce network size in pre-trained LMs, while retaining comparable performance and efficiency. PKD (Sun et al., 2019) was the first known effort in this expedition, an elegant and effective method that uses knowledge distillation (KD) for BERT model compression at finetuning stage. Later on, DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019) and MobileBERT (Sun et al., 2020) car- ried on the torch and extended similar compression techniques to pre-training stage, allowing efficient training of task-agnostic compressed models. In addition to the conventional KL-divergence loss ap- plied to the probabilistic output of the teacher and student networks, an L2 loss measuring the differ- ence between normalized hidden layers has proven to be highly effective in these methods. However, L2 norm follows the assumption that all dimen- sions of the target representation are independent, which overlooks important structural information in the many hidden layers of BERT teacher. Motivated by this, we propose Contrastive Distillation for Intermediate Representations (CODIR), which uses a contrastive objective to capture higher-order output dependencies between intermediate representations of BERT teacher and the student. Contrastive learning (Gutmann and Hyv¨arinen, 2010) aims to learn representations by enforcing similar elements to …

Content truncated to 500 words