Sliced Recurrent Neural Networks

Proceedings of the 27th International Conference on Computational Linguistics, pages 2953–2964 Santa Fe, New Mexico, USA, August 20-26, 2018. 2953 Sliced Recurrent Neural Networks Zeping Yu School of Eclectronic Information and Electrical Engineering Shanghai Jiao Tong University zepingyu@foxmail.com Gongshen Liu* School of Eclectronic Information and Electrical Engineering Shanghai Jiao Tong University lgshen@sjtu.edu.cn Abstract Recurrent neural networks have achieved great success in many NLP tasks. However, they have difficulty in parallelization because of the recurrent structure, so it takes much time to train RNNs. In this paper, we introduce sliced recurrent neural networks (SRNNs), which could be parallelized by slicing the sequences into many subsequences. SRNNs have the ability to obtain high-level information through multiple layers with few extra parameters. We prove that the standard RNN is a special case of the SRNN when we use linear activation functions. Without changing the recurrent units, SRNNs are 136 times as fast as standard RNNs and could be even faster when we train longer sequences. Experiments on six large- scale sentiment analysis datasets show that SRNNs achieve better performance than standard RNNs. 1 Introduction Recurrent neural networks (RNNs) have been widely used in many NLP tasks, including machine translation (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Bradbury and Socher, 2016), question answering (Xiong et al., 2016; Chen et al., 2017), image caption (Xu et al., 2015; Karpathy and Li, 2015), and document classification (Tang et al., 2015; Yang et al., 2016; Zhou et al., 2016). RNNs have the ability to obtain the order information of the input sequences. The two most popular recurrent units are long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and gated recurrent unit (GRU) (Cho et al., 2014), both of which could store previous memory in hidden states and use a gating mechanism to determine how much previous memory should be combined with the current input. Unfortunately, because of the recurrent structure, RNNs cannot be computed in parallel. Therefore, training RNNs takes much time, which limits academic research and industrial applications. To solve this problem, several scholars try to use convolutional neural networks (CNNs) (Lecun et al., 1998) instead of RNNs in the field of NLP (Kim, 2014; Kalchbrenner et al., 2014; Gehring et al., 2017). However, CNNs may not obtain the order information of the sequences, which is very important in NLP tasks. Some scholars tried to increase the speed of RNNs by improving the recurrent units and they have achieved good results. Quasi-recurrent neural networks (QRNNs) (Bradbury et al., 2017) got up to 16 times faster speeds by combining CNNs with RNNs. Lei et al. (2017) proposed the simple recurrent unit (SRU), which is 5-10 times faster than LSTM. Similarly, strongly-typed recurrent neural networks (T-RNN) (Balduzzi and Ghifary, 2016) and minimal gated unit (MGU) (Zhou et al., 2016) are also methods that could change the recurrent units. Although RNNs have achieved faster speeds in these researches with the recurrent units improved, the recurrent structure among the entire sequence remains unchanged. As we …

Content truncated to 500 words