Consistent Accelerated Inference via Confident Adaptive Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4962–4979 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4962 Consistent Accelerated Inference via Confident Adaptive Transformers Tal Schuster∗ Adam Fisch∗ Tommi Jaakkola Regina Barzilay Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {tals,fisch,tommi,regina}@csail.mit.edu Abstract We develop a novel approach for confidently accelerating inference in the large and ex- pensive multilayer Transformers that are now ubiquitous in natural language processing (NLP). Amortized or approximate computa- tional methods increase efficiency, but can come with unpredictable performance costs. In this work, we present CATs—Confident Adaptive Transformers—in which we simul- taneously increase computational efficiency, while guaranteeing a specifiable degree of con- sistency with the original model with high con- fidence. Our method trains additional pre- diction heads on top of intermediate layers, and dynamically decides when to stop allocat- ing computational effort to each input using a meta consistency classifier. To calibrate our early prediction stopping rule, we formulate a unique extension of conformal prediction. We demonstrate the effectiveness of this approach on four classification and regression tasks.1 1 Introduction Large pre-trained language models have become the de facto standard approach for solving natu- ral language processing tasks (Devlin et al., 2019; Liu et al., 2019). Despite their impressive perfor- mance, however, their often massive computational burden makes them costly to run (Schwartz et al., 2019; Sharir et al., 2020). Concerns about their effi- ciency have kindled a large body of research in the field (Sanh et al., 2020; Schwartz et al., 2020; Fan et al., 2020). For multilayered architectures such as the Transformer, a popular approach is adap- tive early exiting (Schwartz et al., 2020; Xin et al., 2020a, inter alia). Early exiting takes advantage of the observation that task instances vary in complex- ity. In this setting, “early” classifiers are added on top of the simpler features of intermediate layers *The first two authors contributed equally. 1� https://github.com/TalSchuster/CATs Figure 1: Our CAT model G can save computational re- sources by exiting early on certain inputs—while guar- anteeing predictive consistency with the full model F. in the base model, and can trigger a prediction be- fore the full model is executed. Naively deciding when to preempt computation, however, can result in unpredictable decreases in model accuracy. Quantifying the uncertainty in a prediction in order to decide when additional computation is needed (or not) is critical to making predic- tions quickly without excessively sacrificing per- formance. In this paper, we present Confident Adaptive Transformers (CATs), a general method for increasing Transformer-based model efficiency while remaining confident in the quality of our pre- dictions. Specifically, given a fixed, expensive l- layer model F(x), we create an amortized model G(x) that includes early classifiers {F1, . . . , Fl}.2 We then make G provably consistent with the origi- nal F with arbitrarily high probability (e.g., 95% of the time). This process is illustrated in Figure 1. Our approach builds on conformal prediction (CP), a model-agnostic and distribution-free frame- work …

Content truncated to 500 words