MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2158 MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun1∗, Hongkun Yu2, Xiaodan Song2, Renjie Liu2, Yiming Yang1, Denny Zhou2 1Carnegie Mellon University {zhiqings, yiming}@cs.cmu.edu 2Google Brain {hongkuny, xiaodansong, renjieliu, dennyzhou}@google.com Abstract Natural Language Processing (NLP) has re- cently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suf- fer from heavy model sizes and high latency such that they cannot be deployed to resource- limited mobile devices. In this paper, we pro- pose MobileBERT for compressing and accel- erating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERTLARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward net- works. To train MobileBERT, we first train a specially designed teacher model, an inverted- bottleneck incorporated BERTLARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical stud- ies show that MobileBERT is 4.3× smaller and 5.5× faster than BERTBASE while achiev- ing competitive results on well-known bench- marks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERTBASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERTBASE). 1 Introduction The NLP community has witnessed a revolution of pre-training self-supervised models. These models usually have hundreds of millions of parameters (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019). Among these models, BERT (Devlin et al., 2018) ∗This work was done when the first author was an intern at Google Brain. shows substantial accuracy improvements. How- ever, as one of the largest models ever in NLP, BERT suffers from the heavy model size and high latency, making it impractical for resource-limited mobile devices to deploy the power of BERT in mobile-based machine translation, dialogue model- ing, and the like. There have been some efforts that task- specifically distill BERT into compact models (Turc et al., 2019; Tang et al., 2019; Sun et al., 2019; Tsai et al., 2019). To the best of our knowl- edge, there is not yet any work for building a task- agnostic lightweight pre-trained model, that is, a model that can be generically fine-tuned on differ- ent downstream NLP tasks as the original BERT does. In this paper, we propose MobileBERT to fill this gap. In practice, task-agnostic compression of BERT is desirable. Task-specific compression needs to first fine-tune the original large BERT model into a task-specific teacher and then distill. Such a process is much more complicated (Wu et al., 2019) and costly than directly fine-tuning a task-agnostic compact model. At first …

Content truncated to 500 words