Recurrent Dropout without Memory Loss

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1757–1766, Osaka, Japan, December 11-17 2016. Recurrent Dropout without Memory Loss Stanislau Semeniuta1 Aliaksei Severyn2 Erhardt Barth1 1Universit¨at zu L¨ubeck, Institut f¨ur Neuro- und Bioinformatik {stas,barth}@inb.uni-luebeck.de 2Google Research severyn@google.com Abstract This paper presents a novel approach to recurrent neural network (RNN) regularization. Dif- ferently from the widely adopted dropout method, which is applied to forward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in recurrent connec- tions in a way that does not cause loss of long-term memory. Our approach is as easy to im- plement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells . Our experiments on three NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout. 1 Introduction Recurrent Neural Networks, LSTMs in particular, have recently become a popular tool among NLP researchers for their superior ability to model and learn from sequential data. These models have shown state-of-the-art results on various public benchmarks ranging from sentence classification (Wang et al., 2015; Irsoy and Cardie, 2014; Liu et al., 2015) and various tagging problems (Dyer et al., 2015) to language modelling (Kim et al., 2015; Zhang et al., 2015), text generation (Zhang and Lapata, 2014) and sequence-to-sequence prediction tasks (Sutskever et al., 2014). Having shown excellent ability to capture and learn complex linguistic phenomena, RNN architectures are prone to overfitting. Among the most widely used techniques to avoid overfitting in neural networks is the dropout regularization (Hinton et al., 2012). Since its introduction it has become, together with the L2 weight decay, the standard method for neural network regularization. While showing significant im- provements when used in feed-forward architectures, e.g., Convolutional Neural Networks (Krizhevsky et al., 2012), the application of dropout in RNNs has been somewhat limited. Indeed, so far dropout in RNNs has been applied in the same fashion as in feed-forward architectures: it is typically injected in input-to-hidden and hidden-to-output connections, i.e., along the input axis, but not between the recur- rent connections (time axis). Given that RNNs are mainly used to model sequential data with the goal of capturing short- and long-term interactions, it seems natural to also regularize the recurrent weights. This observation has led us and other researchers (Moon et al., 2015; Gal, 2015) to the idea of applying dropout to the recurrent connections in RNNs. In this paper we propose a novel recurrent dropout technique and demonstrate how our method is superiour to other recurrent dropout methods recently proposed in (Moon et al., 2015; Gal, 2015). Ad- ditionally, we answer the following questions which helps to understand how to best apply recurrent dropout: (i) how to apply the dropout in recurrent connections of the LSTM architecture in a way that prevents possible corruption of the long-term memory; (ii) what is the relationship between our recurrent dropout and the widely adopted dropout in input-to-hidden and hidden-to-output …

Content truncated to 500 words