E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5229 - 5244 May 22-27, 2022 c⃝2022 Association for Computational Linguistics E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models Mohammad Akbari, Amin Banitalebi-Dehkordi, Yong Zhang Huawei Technologies Canada Co., Ltd. {mohammad.akbari, amin.banitalebi, yong.zhang3}@huawei.com Abstract Building huge and highly capable language models has been a trend in the past years. Despite their great performance, they incur high computational cost. A common solu- tion is to apply model compression or choose light-weight architectures, which often need a separate fixed-size model for each desirable computational budget, and may lose perfor- mance in case of heavy compression. This paper proposes an effective dynamic inference approach, called E-LANG, which distributes the inference between large accurate Super- models and light-weight Swift models. To this end, a decision making module routes the in- puts to Super or Swift models based on the energy characteristics of the representations in the latent space. This method is easily adopt- able and architecture agnostic. As such, it can be applied to black-box pre-trained mod- els without a need for architectural manipula- tions, reassembling of modules, or re-training. Unlike existing methods that are only applica- ble to encoder-only backbones and classifica- tion tasks, our method also works for encoder- decoder structures and sequence-to-sequence tasks such as translation. The E-LANG perfor- mance is verified through a set of experiments with T5 and BERT backbones on GLUE, Su- perGLUE, and WMT. In particular, we out- perform T5-11B with an average computa- tions speed-up of 3.3× on GLUE and 2.9× on SuperGLUE. We also achieve BERT-based SOTA on GLUE with 3.2× less computations. Code and demo are available here. 1 Introduction With the introduction of influential language mod- els such as BERT (Devlin et al., 2019), a trend in natural language processing (NLP) research has been to develop high capacity models and push their performance to new levels. Consequently, state-of-the-art (SOTA) results were achieved on various benchmarks using these models; GPT-3 (Brown et al., 2020), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), ELECTRA (Clark et al., 2020), and DeBERTa (He et al., 2021) to name a few. A potential down-side, however, is that the number of parameters or float- ing point operations (FLOPs) for these models can get extremely large. For example, Gshard (Lep- ikhin et al., 2021) comes with 600B parameters with an enormous amount of computation. This in turn results in a higher inference latency, which is not desirable for latency-sensitive applications. A common solution to speed-up the large lan- guage models is to apply model compression (Gupta et al., 2020). Although generally success- ful, compression does come with a trade-off on accuracy, and may lose performance if compres- sion is heavy. In addition, these methods usually compress a model to a fixed smaller size, where a separate model is required for each possible compu- tational budget. An alternative approach explored in the literature is to …

Content truncated to 500 words