The Right Tool for the Job: Matching Model and Instance Complexities

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640–6651 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6640 The Right Tool for the Job: Matching Model and Instance Complexities Roy Schwartz♦♠ Gabriel Stanovsky♦♠ Swabha Swayamdipta♦ Jesse Dodge♣∗ Noah A. Smith♦♠ ♦Allen Institute for Artificial Intelligence ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ♣School of Computer Science, Carnegie Mellon University {roys,gabis,swabhas,jessed,noah}@allenai.org Abstract As NLP models become larger, executing a trained model requires significant computa- tional resources incurring monetary and envi- ronmental costs. To better respect a given inference budget, we propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) “exit” from neural network calculations for simple instances, and late (and accurate) exit for hard instances. To achieve this, we add classifiers to different layers of BERT and use their calibrated confidence scores to make early exit decisions. We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks. Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing mod- els which are up to five times faster than the state of the art, while preserving their accu- racy. Our method also requires almost no ad- ditional training resources (in either time or parameters) compared to the baseline BERT model. Finally, our method alleviates the need for costly retraining of multiple models at dif- ferent levels of efficiency; we allow users to control the inference speed/accuracy tradeoff using a single trained model, by setting a sin- gle variable at inference time. We publicly re- lease our code.1 1 Introduction The large increase in the size of artificial intel- ligence models often increases production costs (Amodei and Hernandez, 2018; Schwartz et al., 2019), and can also limit adoption on real-time de- vices. Compared to training, which is a one-time large investment, inference costs are incurred for every instance in production, and can thus add up ∗Research completed during an internship at AI2. 1github.com/allenai/sledgehammer Layer 0 Layer i Layer k Layer n Input Layer l Layer j Is confident? Yes No Is confident? Yes No Is confident? Yes No Prediction Early exit prediction Early exit prediction Early exit prediction Figure 1: An illustration of our approach. Some lay- ers of a BERT-large model are attached to output clas- sifiers, which make their respective predictions. The confidence of each layer-wise prediction is computed. If high enough, the model takes an early exit, avoid- ing the computation associated with successive (higher) layers (grayed out). Otherwise, the model continues to the next layer/classifier. significantly. For instance, Microsoft reports that using BERT (Devlin et al., 2019) to process Bing queries requires more than 2,000 GPUs concur- rently.2 We present a method to reduce the inference cost of today’s common models in NLP: fine-tuned con- textual word representations. Our method exploits variation along two axes: models differ in size and cost, and instances vary in difficulty. Our …

Content truncated to 500 words