Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11007 - 11021 December 7-11, 2022 ©2022 Association for Computational Linguistics Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems Neeraj Varshney and Chitta Baral Arizona State University {nvarshn2, cbaral}@asu.edu Abstract Do all instances need inference through the big models for a correct prediction? Perhaps not; some instances are easy and can be answered correctly by even small capacity models. This provides opportunities for im- proving the computational efficiency of sys- tems. In this work, we present an explorative study on ‘model cascading’, a simple technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions. Through comprehensive experi- ments in multiple task settings that differ in the number of models available for cascading (K value), we show that cascading improves both the computational efficiency and the prediction accuracy. For instance, in K=3 setting, cascad- ing saves up to 88.93% computation cost and consistently achieves superior prediction accu- racy with an improvement of up to 2.18%. We also study the impact of introducing additional models in the cascade and show that it further increases the efficiency improvements. Finally, we hope that our work will facilitate develop- ment of efficient NLP systems making their widespread adoption in real-world applications possible. 1 Introduction Pre-trained language models such as RoBERTa (Liu et al., 2019), ELECTRA (Clark et al., 2020), and T5 (Raffel et al., 2020) have achieved remark- able performance on numerous natural language processing benchmarks (Wang et al., 2018, 2019; Talmor et al., 2019). However, these models have a large number of parameters which makes them slow and computationally expensive; for instance, T5-11B requires ∼87 × 1011 floating point oper- ations (FLOPs) for an inference. This limits their widespread adoption in real-world applications that prefer computationally efficient systems in order to achieve low response times. The above concern has recently received consid- erable attention from the NLP community leading Figure 1: Illustrating a cascading approach with three models (Mini, Med, and Base) arranged in increasing order of capacity. An input is first passed through the smallest model (Mini) which fails to predict with suf- ficient confidence. Therefore, it is then inferred using a bigger model (Med) that satisfies the confidence con- straints and the system outputs its prediction (‘contradic- tion’ as dog has four legs). Thus, by avoiding inference through large/expensive models, the system saves com- putation cost without sacrificing the accuracy. to development of several techniques, such as (1) network pruning that progressively removes model weights from a big network (Wang et al., 2020; Guo et al., 2021), (2) early exiting that allows mul- tiple exit paths in a model (Xin et al., 2020), (3) adaptive inference that adjusts model size by adap- tively selecting its width and depth (Goyal et al., 2020; Kim and Cho, 2021), (4) knowledge distilla- tion that transfers ‘dark-knowledge’ from a large teacher model to a shallow student model (Jiao et al., 2020; Li et al., …

Content truncated to 500 words