Attention Mechanism with Energy-Friendly Operations

Findings of the Association for Computational Linguistics: ACL 2022, pages 3969 - 3976 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Attention Mechanism with Energy-Friendly Operations Yu Wana,b∗† Baosong Yangb∗ Dayiheng Liub Rong Xiaob†‡ Derek F. Wonga Haibo Zhangb Boxing Chenb Lidia S. Chaoa aNLP2CT Lab, University of Macau nlp2ct.ywan@gmail.com, {derekfw,lidiasc}@umac.mo bAlibaba Group {yangbaosong.ybs,liudayiheng.ldyh,birong.xr,zhanhui.zhb, boxing.cbx}@alibaba-inc.com Abstract Attention mechanism has become the domi- nant module in natural language processing models. It is computationally intensive and depends on massive power-hungry multipli- cations. In this paper, we rethink variants of attention mechanism from the energy con- sumption aspects. After reaching the conclu- sion that the energy costs of several energy- friendly operations are far less than their multi- plication counterparts, we build a novel atten- tion model by replacing multiplications with either selective operations or additions. Em- pirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure. Code is available at: https://github.com/NLP2CT/E-Att. 1 Introduction Attention mechanism (ATT, Bahdanau et al., 2015; Vaswani et al., 2017; Yang et al., 2018) has demon- strated huge success in a variety of natural language processing tasks (Su et al., 2018; Kitaev and Klein, 2018; Tan et al., 2018; Yang et al., 2019a; Devlin et al., 2019; Zhang et al., 2020). The module learns hidden representations of a sequence by serving each word as a query to attend to all keys in the tar- get sentence, then softly assembling their values. It is a de-facto standard to achieve this via performing linear projections and dot products on representa- tions of queries and keys (Vaswani et al., 2017), re- sulting in large amount of multiplications. In spite of its promising quality, such kind of paradigm may be not the preferred solution from the energy consumption aspect (Horowitz, 2014; Raffel et al., ∗Equal contribution. †Work was done when interning at DAMO Academy, Alibaba Group. ‡Corresponding author. Operation (FP32) ASIC FPGA Addition 0.9 0.4 Multiplication 3.7 18.8 Table 1: Energy cost (10−12Joule) of addition and multiplication on ASIC/FPGA chips (You et al., 2020). The representative of ASIC chip is Google’s TPU, while Microsoft cloud employs FPGA chips. 2020). How to build a high energy-efficient ATT still remains a great challenge. Our work starts from in-depth investigations on approaches in ATT context with respect to model compression (Hinton et al., 2015; Jiao et al., 2020) and complexity optimization (Yang et al., 2019b; Raganato et al., 2020; Beltagy et al., 2020; Tay et al., 2021). These approaches can potentially al- leviate the problem of high energy consumption in ATT. Nevertheless, intentions of all these methods are not exactly from the energy-friendly perspec- tive, thus overlooking the origin of energy con- sumed, i.e., basic arithmetic operations in electric equipments. Massive multiplications still remain, consuming far more energy than its additive coun- terpart on modern devices (Table 1, Li et al., 2020). To this end, we propose to approach this prob- lem …

Content truncated to 500 words