Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 321–331 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1030 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 321–331 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1030 Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling Zhe Gan∗, Chunyuan Li∗†, Changyou Chen, Yunchen Pu, Qinliang Su, Lawrence Carin Department of Electrical and Computer Engineering, Duke University {zg27, cl319, cc448, yp42, qs15, lcarin}@duke.edu Abstract Recurrent neural networks (RNNs) have shown promising performance for lan- guage modeling. However, traditional training of RNNs using back-propagation through time often suffers from overfit- ting. One reason for this is that stochastic optimization (used for large training sets) does not provide good estimates of model uncertainty. This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo (also appropriate for large training sets) to learn weight uncer- tainty in RNNs. It yields a principled Bayesian learning algorithm, adding gra- dient noise during training (enhancing ex- ploration of the model-parameter space) and model averaging when testing. Ex- tensive experiments on various RNN mod- els and across a broad range of applica- tions demonstrate the superiority of the proposed approach relative to stochastic optimization. 1 Introduction Language modeling is a fundamental task, used for example to predict the next word or charac- ter in a text sequence given the context. Recently, recurrent neural networks (RNNs) have shown promising performance on this task (Mikolov et al., 2010; Sutskever et al., 2011). RNNs with Long Short-Term Memory (LSTM) units (Hochre- iter and Schmidhuber, 1997) have emerged as a popular architecture, due to their representational power and effectiveness at capturing long-term de- pendencies. RNNs are usually trained via back-propagation through time (Werbos, 1990), using stochastic op- ∗Equal contribution. †Corresponding author. timization methods such as stochastic gradient de- scent (SGD) (Robbins and Monro, 1951); stochas- tic methods of this type are particularly important for training with large data sets. However, this approach often provides a maximum a posteriori (MAP) estimate of model parameters. The MAP solution is a single point estimate, ignoring weight uncertainty (Blundell et al., 2015; Hern´andez- Lobato and Adams, 2015). Natural language of- ten exhibits significant variability, and hence such a point estimate may make over-confident predic- tions on test data. To alleviate overfitting RNNs, good regular- ization is known as a key factor to successful applications. In the neural network literature, Bayesian learning has been proposed as a princi- pled method to impose regularization and incor- porate model uncertainty (MacKay, 1992; Neal, 1995), by imposing prior distributions on model parameters. Due to the intractability of poste- rior distributions in neural networks, Hamiltonian Monte Carlo (HMC) (Neal, 1995) has been used to provide sample-based approximations to the true posterior. Despite the elegant theoretical prop- erty of asymptotic convergence to the true poste- rior, HMC and other conventional Markov Chain Monte Carlo methods are not scalable to large training …

Content truncated to 500 words