Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 192–202, Osaka, Japan, December 11-17 2016. Asynchronous Parallel Learning for Neural Networks and Structured Models with Dense Features Xu Sun ∗MOE Key Laboratory of Computational Linguistics, Peking University †School of Electronics Engineering and Computer Science, Peking University xusun@pku.edu.cn Abstract Existing asynchronous parallel learning methods are only for the sparse feature models, and they face new challenges for the dense feature models like neural networks (e.g., LSTM, RNN). The problem for dense features is that asynchronous parallel learning brings gradient errors derived from overwrite actions. We show that gradient errors are very common and inevitable. Never- theless, our theoretical analysis shows that the learning process with gradient errors can still be convergent towards the optimum of objective functions for many practical applications. Thus, we propose a simple method AsynGrad for asynchronous parallel learning with gradient error. Base on various dense feature models (LSTM, dense-CRF) and various NLP tasks, experiments show that AsynGrad achieves substantial improvement on training speed, and without any loss on accuracy. 1 Introduction Stochastic learning methods can accelerate the training speed compared with traditional batch training methods. A widely used stochastic learning method is the stochastic gradient descent method (SGD) (Bertsekas, 1999; Bottou and Bousquet, 2008; Shalev-Shwartz and Srebro, 2008; Sun et al., 2012; Sun et al., 2014). For large-scale datasets, the SGD training methods can be much faster than batch training methods. For further improve the training speed over multi-core machines and clusters, a variety of asynchronous (lock-free) parallel learning methods has been developed based on stochastic learning (Niu et al., 2011; Mcmahan and Streeter, 2014). Those asynchronous methods have shown to be more efficient than the synchronous (locked) parallel learning versions (Langford et al., 2009; Gimpel et al., 2010). Other related work on parallel stochastic learning also includes (Zinkevich et al., 2010; Dekel et al., 2012; Recht and Re, 2013; Dean et al., 2012). Existing asynchronous parallel learning methods are mainly for the sparse feature models, and feature sparseness is a major assumption for those parallel learning methods (Niu et al., 2011; Mcmahan and Streeter, 2014). For example, Niu et al. (2011) proposed an interesting asynchronous parallel learning method HogWild for strict sparse machine learning problems with sparse separable cost functions (e.g., sparse SVM, low-rank matrix completion). Niu et al. (2011) stated that the key idea that underlies their lock-free approach is that the targeted machine learning problems are sparse. More recently, Mcmahan and Streeter (2014) proposed a delay-tolerant asynchronous parallel learning method, which is an ex- tension of the method of Niu et al. (2011). This asynchronous parallel learning method proposed by Mcmahan and Streeter (2014) also strictly requires the sparseness of the features. The situation is different for the dense feature models like neural networks (e.g., LSTM, RNN). Since the existing asynchronous parallel learning methods are strictly for sparse feature models, it is no longer reasonable to apply those methods for the dense feature models like neural networks. To our …

Content truncated to 500 words