The Lower The Simpler: Simplifying Hierarchical Recurrent Models

Proceedings of NAACL-HLT 2019, pages 4005–4009 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4005 The Lower The Simpler: Simplifying Hierarchical Recurrent Models Chao Wang and Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering, York University 4700 Keele Street, Toronto, Ontario, Canada {chwang, hj}@eecs.yorku.ca Abstract To improve the training efficiency of hierar- chical recurrent models without compromis- ing their performance, we propose a strategy named as “the lower the simpler”, which is to simplify the baseline models by making the lower layers simpler than the upper layers. We carry out this strategy to simplify two typical hierarchical recurrent models, namely Hier- archical Recurrent Encoder-Decoder (HRED) and R-NET, whose basic building block is GRU. Specifically, we propose Scalar Gated Unit (SGU), which is a simplified variant of GRU, and use it to replace the GRUs at the middle layers of HRED and R-NET. Besides, we also use Fixed-size Ordinally-Forgetting Encoding (FOFE), which is an efficient encod- ing method without any trainable parameter, to replace the GRUs at the bottom layers of HRED and R-NET. The experimental results show that the simplified HRED and the simpli- fied R-NET contain significantly less trainable parameters, consume significantly less train- ing time, and achieve slightly better perfor- mance than their baseline models. 1 Introduction With the advance of various deep learning frame- works, neural network based models proposed for natural language understanding tasks are becom- ing increasingly complicated. To the best of our knowledge, a considerable part of these compli- cated models are both hierarchical and recurrent. For example, Hierarchical Recurrent Encoder- Decoder (HRED) (Sordoni et al., 2015; Serban et al., 2016), which is a conversational model, is constructed by stacking three layers of GRUs (Cho et al., 2014). Besides, several well-known Machine Reading Comprehension (MRC) models, such as R-NET (Wang et al., 2017) and Fusion- Net (Huang et al., 2017), are mainly composed of multiple layers of bidirectional GRUs (BiGRUs) or bidirectional LSTMs (BiLSTMs) (Hochreiter and Schmidhuber, 1997). The above hierarchical recurrent models have achieved excellent perfor- mance, but training them usually consumes a lot of time and memory, that is because their compu- tational graphs contain a large amount of operators and trainable parameters, which makes their train- ing computationally expensive. According to Williams and Zipser (1995), in the training of recurrent neural networks, it is the backward propagation rather than the forward propagation that consumes the majority of the computational resources. Besides, considering the chain rule in the backward propagation, the com- plexity of computing gradients for a hierarchical recurrent model increases exponentially from the top layer of the model down to the bottom layer. Therefore, to improve the training efficiency of hi- erarchical recurrent models, our strategy is to sim- plify the baseline models by making the lower layers simpler than the upper layers, which we name as “the lower the simpler”. Here “sim- pler” means containing less operators and train- able parameters. This strategy is guaranteed to work, since it …

Content truncated to 500 words