One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4408–4413 August 1–6, 2021. ©2021 Association for Computational Linguistics 4408 One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers Chuhan Wu† Fangzhao Wu‡ Yongfeng Huang† †Department of Electronic Engineering & BNRist, Tsinghua University, Beijing 100084, China ‡Microsoft Research Asia, Beijing 100080, China {wuchuhan15, wufangzhao}@gmail.com, yfhuang@tsinghua.edu.cn Abstract Pre-trained language models (PLMs) achieve great success in NLP. However, their huge model sizes hinder their applications in many practical systems. Knowledge distillation is a popular technique to compress PLMs, which learns a small student model from a large teacher PLM. However, the knowledge learned from a single teacher may be limited and even biased, resulting in low-quality student model. In this paper, we propose a multi-teacher knowledge distillation framework named MT- BERT for pre-trained language model com- pression, which can train high-quality student model from multiple teacher PLMs. In MT- BERT we design a multi-teacher co-finetuning method to jointly finetune multiple teacher PLMs in downstream tasks with shared pool- ing and prediction layers to align their output space for better collaborative teaching. In ad- dition, we propose a multi-teacher hidden loss and a multi-teacher distillation loss to transfer the useful knowledge in both hidden states and soft labels from multiple teacher PLMs to the student model. Experiments on three bench- mark datasets validate the effectiveness of MT- BERT in compressing PLMs. 1 Introduction Pre-trained language models (PLMs) such as BERT and RoBERTa have achieved notable success in various NLP tasks (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019). However, many PLMs have a huge model size and computational complexity, making it difficult to deploy them to low-latency and high-concurrence online systems or devices with limited computational resources (Jiao et al., 2020; Wu et al., 2021). Knowledge distillation is a widely used tech- nique for compressing large-scale pre-trained lan- guage models (Sun et al., 2019; Wang et al., 2020). For example, Sanh et al. (2019) proposed Distil- BERT to compress BERT by transferring knowl- edge from the soft labels predicted by the teacher model to student model with a distillation loss. Jiao et al. (2020) proposed TinyBERT, which aligns the hidden states and the attention heatmaps between student and teacher models. These methods usu- ally learn the student model from a single teacher model (Gou et al., 2020). However, the knowledge and supervision provided by a single teacher model may be insufficient to learn an accurate student model, and the student model may also inherit the bias in the teacher model (Bhardwaj et al., 2020). Fortunately, many different large PLMs such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and UniLM (Dong et al., 2019) are off-the- shelf. These PLMs may encode complementary knowledge because they usually have different con- figurations and are trained on different corpus with different self-supervision tasks (Qiu et al., 2020). Thus, incorporating multiple pre-trained language models into knowledge distillation has the potential to learn better student models. In this paper, …

Content truncated to 500 words