Efficient Sequence Learning with Group Recurrent Networks

Proceedings of NAACL-HLT 2018, pages 799–808 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Efficient Sequence Learning with Group Recurrent Networks Fei Gao1,2, Lijun Wu3, Li Zhao2, Tao Qin2, Xueqi Cheng1 and Tie-Yan Liu2 1Institute of Computing Technology, Chinese Academy of Sciences 2Microsoft Research 3School of Data and Computer Science, Sun Yat-sen University {feiga,lizo,taoqin,tie-yan.liu}@microsoft.com; wulijun3@mail2.sysu.edu.cn; cxq@ict.ac.cn Abstract Recurrent neural networks have achieved state-of-the-art results in many artificial in- telligence tasks, such as language modeling, neural machine translation, speech recognition and so on. One of the key factors to these successes is big models. However, training such big models usually takes days or even weeks of time even if using tens of GPU cards. In this paper, we propose an efficient architecture to improve the efficiency of such RNN model training, which adopts the group strategy for recurrent layers, while exploiting the representation rearrangement strategy be- tween layers as well as time steps. To demon- strate the advantages of our models, we con- duct experiments on several datasets and tasks. The results show that our architecture achieves comparable or better accuracy comparing with baselines, with a much smaller number of pa- rameters and at a much lower computational cost. 1 Introduction Recurrent Neural Networks (RNNs) have been widely used for sequence learning, and achieved state-of-the-art results in many artificial intelli- gence tasks in recent years, including language modeling (Zaremba et al., 2014; Merity et al., 2017), neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), and speech recognition (Graves et al., 2013). To get better accuracy, recent state-of-the-art RNN models are designed toward big scale, in- clude going deep (stacking multiple recurrent lay- ers) (Pascanu et al., 2013a) and/or going wide (in- creasing dimensions of hidden states). For exam- ple, an RNN based commercial Neural Machine Translation (NMT) system would employ tens of layers in total, resulting in a large model with hun- dreds of millions of parameters (Wu et al., 2016). However, when the model size increases, the com- putational cost, as well as the memory needed for the training, increases dramatically. The training cost of aforementioned NMT model reaches as high as 1019 FLOPs, and the training procedure spends several days with even 96 GPU cards (Wu et al., 2016) – such complexity is prohibitively ex- pensive. While above models benefit from big neural networks, it is observed that such networks often have redundancy of parameters (Kim and Rush, 2016), motivating us to improve parameter ef- ficiency and design more compact architectures that are more efficient in training while keeping good performance. Recently, many efficient archi- tectures for convolution neural networks (CNNs) have been proposed to reduce training cost in com- puter vision domain. Among them, the group con- volution is one of the most widely used and suc- cessful attempts (Szegedy et al., 2015; Chollet, 2016; Zhang et al., 2017b), which splits the chan- nels into groups and conducts convolution sepa- rately for each group. It’s essentially a diagonal sparse operation …

Content truncated to 500 words