When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7633–7648 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7633 When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute Tao Lei ASAPP, Inc. taoleics@gmail.com Abstract Large language models have become increas- ingly difficult to train because of the grow- ing computation time and cost. In this work, we present SRU++, a highly-efficient archi- tecture that combines fast recurrence and at- tention for sequence modeling. SRU++ ex- hibits strong modeling capacity and training efficiency. On standard language modeling tasks such as ENWIK8, WIKI-103 and BIL- LION WORD datasets, our model obtains bet- ter bits-per-character and perplexity while us- ing 3x-10x less training cost compared to top- performing Transformer models. For instance, our model achieves a state-of-the-art result on the ENWIK8 dataset using 1.6 days of train- ing on an 8-GPU machine. We further demon- strate that SRU++ requires minimal attention for near state-of-the-art performance. Our re- sults suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.1 1 Introduction Many recent advances in language modeling have come from leveraging ever larger datasets and model architectures. As a result, the associated computation cost for developing such models have grown enormously, requiring hundreds of GPU hours or days per experiment, and raising concerns about the environmental sustainability of current research (Schwartz et al., 2020). As a conse- quence, it has become imperative to build compu- tationally efficient models that retain top modeling power while reducing computational costs. The Transformer architecture (Vaswani et al., 2017) was proposed to accelerate model training and has become the predominant architecture in NLP. Specifically, it is built entirely upon self- attention and avoids the use of recurrence to en- able strong parallelization. While this change has 1Our code, experimental setup and models are available at https://github.com/asappresearch/sru. Table 1 Effective GPU hour Transformer-XL SRU++ SRU++ (single attention) 0 1.520 4 1.407 7 1.363 11 1.324 14 1.308 18 1.284 22 1.278 25 1.260 29 1.250 32 1.240 36 1.239 40 1.226 43 1.222 47 1.214 50 1.209 54 1.201 58 1.196 61 1.197 65 1.194 68 1.190 72 1.188 76 1.181 79 1.181 83 1.175 86 1.170 90 1.174 94 1.167 97 1.166 101 1.162 104 1.158 108 1.157 112 1.155 115 1.151 119 1.150 122 1.148 126 1.144 130 1.147 133 1.142 137 1.140 140 1.139 144 1.138 148 1.133 151 1.129 155 1.134 158 1.127 162 1.130 166 1.128 169 1.122 173 1.121 176 1.125 180 1.121 184 1.120 187 1.120 191 1.118 194 1.114 198 1.112 202 1.114 205 1.113 209 1.110 212 1.111 216 1.107 220 1.109 223 1.108 227 1.108 230 1.106 234 1.104 237 1.102 241 1.104 245 1.098 248 1.102 252 1.099 255 1.099 259 1.099 263 1.100 266 1.098 270 1.100 273 1.097 277 1.096 281 1.097 284 1.095 288 1.097 291 1.097 295 1.097 299 1.095 302 1.095 306 1.095 309 1.095 313 1.093 317 1.093 320 …

Content truncated to 500 words