Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3519–3524 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3519 Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change Hongfei Xu1,2 Josef van Genabith1,2 Deyi Xiong3 Qiuhui Liu4∗ 1Saarland University / Saarland, Germany 2German Research Center for Artificial Intelligence / Saarland, Germany 3Tianjin University / Tianjin, China 4China Mobile Online Services / Henan, China hfxunlp@foxmail.com, Josef.Van Genabith@dfki.de, dyxiong@tju.edu.cn, liuqhano@foxmail.com Abstract The choice of hyper-parameters affects the per- formance of neural models. While much pre- vious research (Sutskever et al., 2013; Duchi et al., 2011; Kingma and Ba, 2015) focuses on accelerating convergence and reducing the ef- fects of the learning rate, comparatively few papers concentrate on the effect of batch size. In this paper, we analyze how increasing batch size affects gradient direction, and propose to evaluate the stability of gradients with their an- gle change. Based on our observations, the an- gle change of gradient direction first tends to stabilize (i.e. gradually decrease) while accu- mulating mini-batches, and then starts to fluc- tuate. We propose to automatically and dynam- ically determine batch sizes by accumulating gradients of mini-batches and performing an optimization step at just the time when the di- rection of gradients starts to fluctuate. To im- prove the efficiency of our approach for large models, we propose a sampling approach to select gradients of parameters sensitive to the batch size. Our approach dynamically deter- mines proper and efficient batch sizes during training. In our experiments on the WMT 14 English to German and English to French tasks, our approach improves the Transformer with a fixed 25k batch size by +0.73 and +0.82 BLEU respectively. 1 Introduction The performance of neural models is likely to be affected by the choice of hyper-parameters. While much previous research (Sutskever et al., 2013; Duchi et al., 2011; Kingma and Ba, 2015) focuses on accelerating convergence and reducing the ef- fects of the learning rate, comparatively few papers concentrate on the effect of batch size. However, batch size is also an important hyper- parameter, and some batch sizes empirically lead to better performance than the others. ∗ Corresponding author. Specifically, it has been shown that the perfor- mance of the Transformer model (Vaswani et al., 2017) for Neural Machine Translation (NMT) (Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) relies heavily on the batch size (Popel and Bojar, 2018; Ott et al., 2018; Abdou et al., 2017; Zhang et al., 2019a). The influence of batch size on performance raises the question, how to dynamically find proper and efficient batch sizes during training? In this paper, we investigate the relationship between the batch size and gradients, and propose a dynamic batch size approach by monitoring gradient direc- tion changes. Our contributions are as follows: • We observe the effects on gradients with in- creasing batch size, and find that a large batch size stabilizes the direction of gradients; • We propose to …

Content truncated to 500 words