Understanding the Difficulty of Training Transformers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5747–5763, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5747 Understanding the Difficulty of Training Transformers Liyuan Liu†‡ Xiaodong Liu‡ Jianfeng Gao‡ Weizhu Chen§ Jiawei Han† {ll2, hanj}@illinois.edu , {xiaodl,jfgao,wzchen}@microsoft.com †University of Illinois at Urbana-Champaign ‡Microsoft Research § Microsoft Dynamics 365 AI Abstract Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding carefully design- ing cutting-edge optimizers and learning rate schedulers (e.g., conventional SGD fails to train Transformers effectively). Our objective here is to understand what complicates Trans- former training from both empirical and theo- retical perspectives. Our analysis reveals that unbalanced gradients are not the root cause of the instability of training. Instead, we identify an amplification effect that influences training substantially–for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output. Yet we ob- serve that a light dependency limits the model potential and leads to inferior trained models. Inspired by our analysis, we propose Admin (Adaptive model initialization) to stabilize the early stage’s training and unleash its full po- tential in the late stage. Extensive experiments show that Admin is more stable, converges faster, and leads to better performance1. 1 Introduction Transformers (Vaswani et al., 2017) have led to a series of breakthroughs in various deep learning tasks (Devlin et al., 2019; Velickovic et al., 2018). They do not contain recurrent connections and can parallelize all computations in the same layer, thus improving effectiveness, efficiency, and scal- ability. Training Transformers, however, requires extra efforts. For example, although stochas- tic gradient descent (SGD) is the standard algo- rithm for conventional RNNs and CNNs, it con- verges to bad/suspicious local optima for Trans- 1Implementations are released at: https://github. com/LiyuanLucasLiu/Transforemr-Clinic 0 50 100 4.4 4.6 4.8 5.0 Dev PPL on WMT’14 En-De 18-Layer Transformer 50 100 4.6 4.7 4.8 4.9 6-layer Post-LN converges, but 18-layer Post-LN does not. 6-Layer Transformer Pre-LN Post-LN Admin (Post-LN) Epoch # (iterations over the training set) Figure 1: Lacking enough robustness and stability, the 18-Layer Post-LN Transformer training (i.e.the original architecture) diverges and is omitted in the left graph. Admin not only stabilizes model training but unleashes the model potential for better performance. formers (Zhang et al., 2019b). Moreover, com- paring to other neural architectures, removing the warmup stage in Transformer training results in more severe consequences such as model diver- gence (Popel and Bojar, 2018; Liu et al., 2020a). Here, we conduct comprehensive analyses in em- pirical and theoretical manners to answer the ques- tion: what complicates Transformer training. Our analysis starts from the observation: the original Transformer (referred to as Post-LN) is less robust than its Pre-LN variant2 (Baevski and Auli, 2019; Xiong et al., 2019; Nguyen and Salazar, 2019). We recognize that gradient van- ishing issue is not the direct reason causing such difference, since fixing this issue alone …

Content truncated to 500 words