TinyBERT: Distilling BERT for Natural Language Understanding

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4163 TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao1∗†, Yichun Yin2∗‡, Lifeng Shang2‡, Xin Jiang2 Xiao Chen2, Linlin Li3, Fang Wang1‡ and Qun Liu2 1Key Laboratory of Information Storage System, Huazhong University of Science and Technology, Wuhan National Laboratory for Optoelectronics 2Huawei Noah’s Ark Lab 3Huawei Technologies Co., Ltd. {jiaoxiaoqi,wangfang}@hust.edu.cn {yinyichun,shang.lifeng,jiang.xin}@huawei.com {chen.xiao2,lynn.lilinlin,qun.liu}@huawei.com Abstract Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usu- ally computationally expensive, so it is diffi- cult to efficiently execute them on resource- restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Trans- former distillation method that is specially de- signed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge en- coded in a large “teacher” BERT can be ef- fectively transferred to a small “student” Tiny- BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which per- forms Transformer distillation at both the pre- training and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domain as well as the task-specific knowledge in BERT. TinyBERT41 with 4 layers is empirically ef- fective and achieves more than 96.8% the per- formance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT4 is also signifi- cantly better than 4-layer state-of-the-art base- lines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them. Moreover, TinyBERT6 with 6 layers performs on-par with its teacher BERTBASE. 1 Introduction Pre-training language models then fine-tuning on downstream tasks has become a new paradigm for ∗Authors contribute equally. †This work is done when Xiaoqi Jiao is an intern at Huawei Noah’s Ark Lab. ‡Corresponding authors. 1The code and models are publicly available at https: //github.com/huawei-noah/Pretrained-Language-Model/tree/ master/TinyBERT natural language processing (NLP). Pre-trained lan- guage models (PLMs), such as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2020), T5 (Raf- fel et al., 2019) and ELECTRA (Clark et al., 2020), have achieved great success in many NLP tasks (e.g., the GLUE benchmark (Wang et al., 2018) and the challenging multi-hop reasoning task (Ding et al., 2019)). However, PLMs usually have a large number of parameters and take long infer- ence time, which are difficult to be deployed on edge devices such as mobile phones. Recent stud- ies (Kovaleva et al., 2019; Michel et al., 2019; Voita et al., 2019) demonstrate that there is redundancy in PLMs. Therefore, it is crucial and feasible to reduce the computational overhead and model stor- age of PLMs while retaining their performances. There have been many model compression tech- niques (Han et al., 2016) proposed to accelerate deep model inference and reduce model size while maintaining accuracy. The most commonly used techniques include …

Content truncated to 500 words