IrEne: Interpretable Energy Prediction for Transformers

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2145–2157 August 1–6, 2021. ©2021 Association for Computational Linguistics 2145 IrEne: Interpretable Energy Prediction for Transformers Qingqing Cao, Yash Kumar Lal, Harsh Trivedi, Aruna Balasubramanian, Niranjan Balasubramanian Department of Computer Science Stony Brook University Stony Brook, NY 11794, USA {qicao,ylal,hjtrivedi,arunab,niranjan}@cs.stonybrook.edu Abstract Existing software-based energy measurements of NLP models are not accurate because they do not consider the complex interactions be- tween energy consumption and model execu- tion. We present IrEne, an interpretable and extensible energy prediction system that accu- rately predicts the inference energy consump- tion of a wide range of Transformer-based NLP models. IrEne constructs a model tree graph that breaks down the NLP model into modules that are further broken down into low-level machine learning (ML) primitives. IrEne predicts the inference energy consump- tion of the ML primitives as a function of generalizable features and fine-grained run- time resource usage. IrEne then aggregates these low-level predictions recursively to pre- dict the energy of each module and finally of the entire model. Experiments across multiple Transformer models show IrEne predicts infer- ence energy consumption of transformer mod- els with an error of under 7% compared to the ground truth. In contrast, existing energy mod- els see an error of over 50%. We also show how IrEne can be used to conduct energy bot- tleneck analysis and to easily evaluate the en- ergy impact of different architectural choices. We release the code and data at https:// github.com/StonyBrookNLP/irene. 1 Introduction Accurately measuring the energy consumption of NLP models is becoming ever more important. Models are growing exponentially, with billions, even approaching trillions, of parameters with correspondingly large resource consumption (e.g. GPT-3 (Brown et al., 2020) has 175 billion param- eters and Switch Transformers can have 1.6 trillion parameters (Fedus et al., 2021)). Recent works have sought to estimate energy consumption and suggest ways to reduce the resulting costs and car- bon impacts (Strubell et al., 2019; Schwartz et al., 2019; Henderson et al., 2020; Anthony et al., 2020) Unfortunately, there are no easy-to-use and ac- curate solutions for measuring or predicting the energy consumption. On the one hand, measur- ing energy consumption directly through hardware power monitors is not feasible as it requires ex- clusive access to the hardware and detailed instru- mentation. On the other hand, there are software models that predict energy as a function of resource utilization (Strubell et al., 2019; Henderson et al., 2020) but these energy prediction models are in- accurate (Cao et al., 2020). The inaccuracy stems from the prediction models not accounting for the complex interactions between energy consumption and resource utilization. In this work, we focus on inference energy which can incur substantial costs especially for models that support high-volume web services. We ask how we can build an energy prediction method that is accurate, interpretable, and extensible. We make three contributions in answering this question. First, we frame the …

Content truncated to 500 words