Recurrent Residual Learning for Sequence Classification

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 938–943, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Recurrent Residual Learning for Sequence Classification Yiren Wang∗ University of Illinois at Urbana-Champaign yiren@illinois.edu Fei Tian Microsoft Research fetia@microsoft.com Abstract In this paper, we explore the possibility of leveraging Residual Networks (ResNet), a powerful structure in constructing extremely deep neural network for image understanding, to improve recurrent neural networks (RNN) for modeling sequential data. We show that for sequence classification tasks, incorporat- ing residual connections into recurrent struc- tures yields similar accuracy to Long Short Term Memory (LSTM) RNN with much fewer model parameters. In addition, we propose two novel models which combine the best of both residual learning and LSTM. Experi- ments show that the new models significantly outperform LSTM. 1 Introduction Recurrent Neural Networks (RNNs) are powerful tools to model sequential data. Among various RNN models, Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) is one of the most effective structures. In LSTM, gating mech- anism is used to control the information flow such that gradient vanishing problem in vanilla RNN is better handled, and long range dependency is bet- ter captured. However, as empirically verified by previous works and our own experiments, to obtain fairly good results, training LSTM RNN needs care- fully designed optimization procedure (Hochreiter et al., 2001; Pascanu et al., 2013; Dai and Le, 2015; Laurent et al., 2015; He et al., 2016; Arjovsky et ∗This work was done when the author was visiting Mi- crosoft Research Asia. al., 2015), especially when faced with unfolded very deep architectures for fairly long sequences (Dai and Le, 2015). From another perspective, for constructing very deep neural networks, recently Residual Networks (ResNet) (He et al., 2015) have shown their ef- fectiveness in quite a few computer vision tasks. By learning a residual mapping between layers with identity skip connections (Jaeger et al., 2007), ResNet ensures a fluent information flow, leading to efficient optimization for very deep structures (e.g., with hundreds of layers). In this paper, we explore the possibilities of leveraging residual learning to improve the performances of recurrent structures, in particular, LSTM RNN, in modeling fairly long se- quences (i.e., whose lengths exceed 100). To sum- marize, our main contributions include: 1. We introduce residual connecting mechanism into the recurrent structure and propose recur- rent residual networks for sequence learning. Our model achieves similar performances to LSTM in text classification tasks, whereas the number of model parameters is greatly reduced. 2. We present in-depth analysis of the strengths and limitations of LSTM and ResNet in respect of sequence learning. 3. Based on such analysis, we further propose two novel models that incorporate the strengths of the mechanisms behind LSTM and ResNet. We demonstrate that our models outperform LSTM in many sequence classification tasks. 938 2 Background RNN models sequences by taking sequential input x = {x1, · · · , xT } and generating T step hidden states h = {h1, …

Content truncated to 500 words