LeeBERT: Learned Early Exit for BERT with cross-level optimization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2968–2980 August 1–6, 2021. ©2021 Association for Computational Linguistics 2968 LeeBERT: Learned Early Exit for BERT with Cross-Level Optimization Wei Zhu1,2 ∗ 1 East China Normal University, Shanghai, China 2 Pingan Health Tech, Shanghai, China Abstract Pre-trained language models like BERT are performant in a wide range of natural language tasks. However, they are resource exhaus- tive and computationally expensive for indus- trial scenarios. Thus, early exits are adopt- ed at each layer of BERT to perform adap- tive computation by predicting easier samples with the first few layers to speed up the in- ference. In this work, to improve efficiency without performance drop, we propose a novel training scheme called Learned Early Exiting for BERT (LeeBERT). First, we ask each exit to learn from each other, rather than learning only from the last layer. Second, the weight- s of different loss terms are learned, thus bal- ancing off different objectives. We formulate the optimization of LeeBERT as a bi-level op- timization problem, and we propose a novel cross-level optimization (CLO) algorithm to improve the optimization results. Experiments on the GLUE benchmark show that our pro- posed methods improve the performance of the state-of-the-art (SOTA) early exiting meth- ods for pre-trained models. 1 Introduction The last couple of years have witnessed the rise of pre-trained language models (PLMs), such as BERT (Devlin et al., 2018), GPT (Radford et al., 2019), XLNet (Yang et al., 2019), and ALBERT (Lan et al., 2020), etc. By pre-training on the un- labeled corpus and fine-tuning on labeled ones, BERT-like models achieved considerable improve- ments in many Natural Language Processing (NLP) tasks, such as text classification and natural lan- guage inference (NLI), sequence labeling, etc. However, these PLMs suffer from two problems. The first problem is efficiency. The state-of-the-art (SOTAs) achievements of these models usually rely ∗Contact: 52205901018@stu.ecnu.edu.cn. on very deep model architectures accompanied by high computational demands, impairs their prac- ticalities. Like general search engines or online medical consultation services, industrial settings process generally millions of requests per minute. What makes efficiency more critical is that the traf- fic of online services varies drastically with time. For example, during the flu season, the search re- quests of Dingxiangyuan1 are ten times more than usual. And the number of claims during the holi- days is five to ten times more than that of the work- days for online shopping. Many servers need to be deployed to enable BERT in industrial settings, which is unbearable for many companies. Second, previous literature (Fan et al., 2020; Michel et al., 2019; Zhou et al., 2020) pointed out that large PLMs with dozens of stacked Trans- former layers are over-parameterized and could suf- fer from the “overthinking” problem (Kaya et al., 2019). That is, for many input samples, their shal- low representations at a shallow layer are enough to make a correct classification. In contrast, the …

Content truncated to 500 words