COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1719 - 1730 December 7-11, 2022 ©2022 Association for Computational Linguistics COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models Bowen Shen1,2, Zheng Lin1,2∗, Yuanxin Liu1,3, Zhengxiao Liu1, Lei Wang1∗, Weiping Wang1 1Institute of Information Engineering, Chinese Academy of Sciences 2School of Cyber Security, University of Chinese Academy of Sciences 3MOE Key Laboratory of Computational Linguistics, Peking University {shenbowen, linzheng, liuzhengxiao, wanglei, wangweiping}@iie.ac.cn liuyuanxin@stu.pku.edu.cn Abstract Transformer-based pre-trained language mod- els (PLMs) mostly suffer from excessive over- head despite their advanced capacity. For resource-constrained devices, there is an ur- gent need for a spatially and temporally effi- cient model which retains the major capacity of PLMs. However, existing statically com- pressed models are unaware of the diverse complexities between input instances, poten- tially resulting in redundancy and inadequacy for simple and complex inputs. Also, minia- ture models with early exiting encounter chal- lenges in the trade-off between making predic- tions and serving the deeper layers. Motivated by such considerations, we propose a collab- orative optimization for PLMs that integrates static model compression and dynamic infer- ence acceleration. Specifically, the PLM is slenderized in width while the depth remains intact, complementing layer-wise early exiting to speed up inference dynamically. To address the trade-off of early exiting, we propose a joint training approach that calibrates slender- ization and preserves contributive structures to each exit instead of only the final layer. Ex- periments are conducted on GLUE benchmark and the results verify the Pareto optimality of our approach at high compression and acceler- ation rate with 1/8 parameters and 1/19 FLOPs of BERT. 1 Introduction Pre-training generalized language models and fine- tuning them on specific downstream tasks has be- come the dominant paradigm in natural language processing (NLP) since the advent of Transform- ers (Vaswani et al., 2017) and BERT (Devlin et al., 2019). However, pre-trained language models (PLMs) are predominantly designed to be vast in the pursuit of model capacity and generaliza- tion. With such a concern, the model storage and ∗ Zheng Lin and Lei Wang are the corresponding authors. Tfm Layer 1 Tfm Layer 6 … Emb Layer Clf 1 Clf 6 Clf 12 Tfm Layer 12 … Clf BERTBase COST-EFF (ours) Tfm Layer 12 Tfm Layer 6 Tfm Layer 1 Emb Layer Distillation Dynamic acceleration for temporal efficiency Static slenderization for spatial efficiency Computation demands … … Figure 1: An illustration of COST-EFF model structure and inference procedure. Emb, Tfm and Clf are abbre- viations of embedding, Transformer and classifier, re- spectively. Blue bar charts denote probability distribu- tion output by classifiers. inference time of PLMs are usually high, mak- ing them challenging to be deployed on resource- constrained devices (Sun et al., 2020). Recent studies indicate that Transformer-based PLMs bear redundancy spatially and temporally which comes from the excessive width and depth of the model (Michel et al., 2019; Xin et al., 2021). With static compression methods including net- work pruning (Xia …

Content truncated to 500 words