Towards Efficient NLP: A Standard Evaluation and A Strong Baseline

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3288 - 3303 July 10-15, 2022 ©2022 Association for Computational Linguistics Towards Efficient NLP: A Standard Evaluation and A Strong Baseline Xiangyang Liu1,2∗, Tianxiang Sun1,2∗, Junliang He1,2, Jiawen Wu1,2, Lingling Wu1,2, Xinyu Zhang3, Hao Jiang3, Zhao Cao3, Xuanjing Huang1,2, Xipeng Qiu1,2 † 1School of Computer Science, Fudan University 2Shanghai Key Laboratory of Intelligent Information Processing, Fudan University 3Huawei Poisson Lab {xiangyangliu20,txsun19,xjhuang,xpqiu}@fudan.edu.cn {zhangxinyu35,jianghao66,caozhao1}@huawei.com Abstract Supersized pre-trained language models have pushed the accuracy of various natural lan- guage processing (NLP) tasks to a new state- of-the-art (SOTA). Rather than pursuing the reachless SOTA accuracy, more and more re- searchers start paying attention to model ef- ficiency and usability. Different from accu- racy, the metric for efficiency varies across dif- ferent studies, making them hard to be fairly compared. To that end, this work presents ELUE (Efficient Language Understanding Evaluation), a standard evaluation, and a public leaderboard for efficient NLP models. ELUE is dedicated to depicting the Pareto Frontier for various language understanding tasks, such that it can tell whether and how much a method achieves Pareto improvement. Along with the benchmark, we also release a strong baseline, ElasticBERT, which allows BERT to exit at any layer in both static and dy- namic ways. We demonstrate the ElasticBERT, despite its simplicity, outperforms or performs on par with SOTA compressed and early exit- ing models. With ElasticBERT, the proposed ELUE has a strong Pareto Frontier and makes a better evaluation for efficient NLP models. 1 Introduction Driven by the large-scale pre-training, today’s NLP models have become much more powerful (Devlin et al., 2019; Yang et al., 2019; Lan et al., 2020; Raf- fel et al., 2020; Sun et al., 2020; Brown et al., 2020; Qiu et al., 2020). As a consequence of this drastic increase in performance, these pre-trained language models (PLMs) are notorious for becoming more and more computationally expensive due to the in- creasing number of parameters. Therefore, rather than pre-training a larger model to achieve a new state-of-the-art (SOTA) accuracy, most studies are pursuing improvement on other dimensions such ∗Equal contribution. †Corresponding author. Figure 1: An illustration to show our motivation, that is, building the Pareto frontier can help recognizing whether and how much a method achieves Pareto im- provement. as the number of parameters or FLOPs (Gordon et al., 2020; Sanh et al., 2019; Jiao et al., 2020; Lan et al., 2020; Shen et al., 2020). For these works, the goal has shifted from simple SOTA to "Pareto SOTA". A Pareto SOTA model means that there is no other model is currently better than it on all the dimensions of interest. For example, a model may claim to be Pareto SOTA as long as it achieves the best accuracy under the same number of param- eters or FLOPs. For these efficient models with fewer parameters or FLOPs, it is unfair to get them evaluated on the accuracy-centric benchmarks such as …

Content truncated to 500 words