On the Transformer Growth for Progressive BERT Training

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5174–5180 June 6–11, 2021. ©2021 Association for Computational Linguistics 5174 On the Transformer Growth for Progressive BERT Training Xiaotao Gu␆␄∗ Liyuan Liu␆ Hongkun Yu␄ Jing Li␄ Chen Chen␄ Jiawei Han␆ ␄ Google Research {hongkuny,jingli,chendouble}@google.com ␆ University of Illinois at Urbana-Champaign {xiaotao2,ll2,hanj}@illinois.edu Abstract Due to the excessive cost of large-scale lan- guage model pre-training, considerable efforts have been made to train BERT progressively— start from an inferior but low-cost model and gradually grow the model to increase the computational complexity. Our objective is to advance the understanding of Transformer growth and discover principles that guide pro- gressive training. First, we find that similar to network architecture search, Transformer growth also favors compound scaling. Specifi- cally, while existing methods only conduct net- work growth in a single dimension, we observe that it is beneficial to use compound growth op- erators and balance multiple dimensions (e.g., depth, width, and input length of the model). Moreover, we explore alternative growth oper- ators in each dimension via controlled compar- ison to give operator selection practical guid- ance. In light of our analyses, the proposed method CompoundGrow speeds up BERT pre- training by 73.6% and 82.2% for the base and large models respectively, while achieving comparable performances1. 1 Introduction Thanks to the rapid increase of computing power, large-scale pre-training has been breaking the glass ceiling for natural language processing tasks (Liu et al., 2018; Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020). However, with great power comes great challenges: the required excessive computational consumption significantly impedes the efficient iteration of both research ex- ploration and industrial application. To lower the training cost, many attempts have been made to conduct progressive training, which starts from training an inferior but low-cost model, and gradu- ally increases its resource consumption (Gong et al., ∗Work done while interning at Google. Corresponding Author: Hongkun Yu and Xiaotao Gu. 1Code will be released at: https://github.com/google- research/google-research/tree/master/grow_bert 2019; Devlin et al., 2019). As elaborated in Sec- tion 5, two components are typically needed for de- signing such progressive training algorithms—the growth scheduler and the growth operator (Dong et al., 2020). The former controls when to conduct network growth, and the latter controls how to per- form network growth. Here, our objectives are to better understand growth operators with a focus on Transformer models (Vaswani et al., 2017; Liu et al., 2020b), and specifically, to help design better progressive algorithms for BERT pre-training (De- vlin et al., 2019). Specifically, we recognize the importance of using compound growth operators in our study, which balance different model dimen- sions (e.g., number of layers, the hidden size, and the input sequence length). Regarding previous efforts made on Transformer growth, they mainly focus on one single model di- mension: either the length (Devlin et al., 2019) or the depth (Gong et al., 2019). In this work, how- ever, we find …

Content truncated to 500 words