Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3626–3631, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3626 Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training Alham Fikri Aji Kenneth Heafield Nikolay Bogoychev School of Informatics, University of Edinburgh 10 Crichton Street Edinburgh EH8 9AB Scotland, European Union {a.fikri, n.bogoych}@ed.ac.uk, kheafiel@inf.ed.ac.uk Abstract One way to reduce network traffic in multi- node data-parallel stochastic gradient descent is to only exchange the largest gradients. How- ever, doing so damages the gradient and de- grades the model’s performance. Tranformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient qual- ity by combining the compressed global gradi- ent with the node’s locally computed uncom- pressed gradient. Neural machine translation experiments show that Transformer conver- gence is restored while RNNs converge faster. With our method, training on 4 nodes con- verges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single- node training. 1 Introduction In recent years, neural network models have grown dramatically in terms of number of pa- rameters (Wen et al., 2017), so exchanging gradi- ents during data-parallel training is costly in terms of both bandwidth and time, especially in a dis- tributed setting. Communication can be reduced (possibly at the expense of convergence) by send- ing only the top 1% of largest gradients in terms of absolute values, a method known as gradient drop- ping (Strom, 2015; Dryden et al., 2016; Aji and Heafield, 2017; Lin et al., 2018). Related meth- ods are synchronizing less often (McMahan et al., 2017; Ott et al., 2018; Bogoychev et al., 2018) and quantization (Seide et al., 2014; Alistarh et al., 2016). As these compression methods are lossy, each node’s locally computed gradient is not immedi- ately reflected in the global gradient. Our exper- iments show that gradient compression damages the model’s performance, especially in the case of a Transformer model (Vaswani et al., 2017), which is known to be sensitive to noisy gradients (Chen et al., 2018; Ott et al., 2018; Aji and Heafield, 2019). We aim to repair the compressed gradi- ent by combining it with the local gradients to im- prove the trade-off between convergence and com- pression rates. In this paper, we apply gradient dropping to reduce the inter-node communication during dis- tributed neural network training, which leads to faster training speed but reduced model conver- gence rate. We find that combining the sparse global gradient with the dense local gradient im- proves convergence. However, adding local infor- mation means that nodes’ parameters will diverge over time. We address this by periodically averag- ing the model (McMahan et al., 2017), achieving faster end-to-end training time. 2 Related Work 2.1 Sparse Gradient Compression Gradients are skewed: most values are near zero while very few have large absolute value (Aji and Heafield, 2017). Formally, Pearson’s skew- ness coefficient is typically 2–4, but …

Content truncated to 500 words