BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3009–3018, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3009 BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance Jianquan Li1∗, Xiaokang Liu1∗, Honghong Zhao1, Ruifeng Xu2, Min Yang3†, Yaohong Jin1 1Beijing Ultrapower Software Co.,Ltd., China 2Harbin Institute of Technology (Shenzhen), China 3Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China {lijianquan2,liuxiaokang1,zhaohonghong1,jinyaohong}@ultrapower.com.cn xuruifeng@hit.edu.cn, min.yang@siat.ac.cn Abstract Pre-trained language models (e.g., BERT) have achieved significant success in vari- ous natural language processing (NLP) tasks. However, high storage and computational costs obstruct pre-trained language mod- els to be effectively deployed on resource- constrained devices. In this paper, we propose a novel BERT distillation method based on many-to-many layer mapping, which allows each intermediate student layer to learn from any intermediate teacher layers. In this way, our model can learn from different teacher lay- ers adaptively for various NLP tasks. In ad- dition, we leverage Earth Mover’s Distance (EMD) to compute the minimum cumulative cost that must be paid to transform knowl- edge from teacher network to student net- work. EMD enables the effective matching for many-to-many layer mapping. Further- more, we propose a cost attention mechanism to learn the layer weights used in EMD au- tomatically, which is supposed to further im- prove the model’s performance and accelerate convergence time. Extensive experiments on GLUE benchmark demonstrate that our model achieves competitive performance compared to strong competitors in terms of both accu- racy and model compression. For reproducibil- ity, we release the code and data at https: //github.com/lxk00/BERT-EMD. 1 Introduction In recent years, pre-trained language models, such as GPT (Radford et al., 2018), BERT (Devlin et al., 2018), XL-Net (Yang et al., 2019), have been proposed and applied to many NLP tasks, yielding state-of-the-art performances. However, the promising results of the pre-trained language models come with the high costs of computation ∗ Equal contribution † Min Yang is corresponding author and memory in inference, which obstruct these pre-trained language models to be deployed on resource-constrained devices and real-time applica- tions. For example, the original BERT-base model, which achieved great success in many NLP tasks, has 12 layers and about 110 millions parameters. It is therefore critical to effectively accelerate inference time and reduce the computational work- load while maintaining accuracy. This research issue has attracted increasing attention (Wang et al., 2019; Shen et al., 2019; Tang et al., 2019), of which knowledge distillation (Tang et al., 2019) is consid- ered to be able to provide a practical way. Typically, knowledge distillation techniques train a compact and shallow student network under the guidance of a complicated larger teacher network with a teacher-student strategy (Watanabe et al., 2017). Once trained, this compact student network can be directly deployed in real-life applications. So far, there have been several studies, such as DistilBERT (Tang et al., 2019), BERT-PKD (Sun et al., 2019), TinyBERT (Jiao et al., 2019), which attempt to compress the original BERT …

Content truncated to 500 words