A Lightweight Recurrent Network for Sequence Modeling

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1538–1548 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1538 A Lightweight Recurrent Network for Sequence Modeling Biao Zhang1 Rico Sennrich1,2 1School of Informatics, University of Edinburgh B.Zhang@ed.ac.uk, rico.sennrich@ed.ac.uk 2Institute of Computational Linguistics, University of Zurich Abstract Recurrent networks have achieved great suc- cess on various sequential tasks with the as- sistance of complex recurrent units, but suffer from severe computational inefficiency due to weak parallelization. One direction to allevi- ate this issue is to shift heavy computations outside the recurrence. In this paper, we pro- pose a lightweight recurrent network, or LRN. LRN uses input and forget gates to handle long-range dependencies as well as gradient vanishing and explosion, with all parameter- related calculations factored outside the recur- rence. The recurrence in LRN only manipu- lates the weight assigned to each token, tightly connecting LRN with self-attention networks. We apply LRN as a drop-in replacement of ex- isting recurrent units in several neural sequen- tial models. Extensive experiments on six NLP tasks show that LRN yields the best running efficiency with little or no loss in model per- formance.1 1 Introduction Various natural language processing (NLP) tasks can be categorized as sequence modeling tasks, where recurrent networks (RNNs) are widely ap- plied and contribute greatly to state-of-the-art neu- ral systems (Yang et al., 2018; Peters et al., 2018; Zhang et al., 2018; Chen et al., 2018; Kim et al., 2019). To avoid the optimization bottle- neck caused by gradient vanishing and/or explo- sion (Bengio et al., 1994), Hochreiter and Schmid- huber (1997) and Cho et al. (2014) develop gate structures to ease information propagation from distant words to the current position. Neverthe- less, integrating these traditional gates inevitably increases computational overhead which is accu- mulated along token positions due to the sequen- 1Source code is available at https://github.com/ bzhangGo/lrn. tial nature of RNNs. As a result, the weak par- allelization of RNNs makes the benefits from im- proved model capacity expensive in terms of com- putational efficiency. Recent studies introduce different solutions to this issue. Zhang et al. (2018) introduce the addition-subtraction twin-gated recurrent unit (ATR), which reduces the amount of matrix oper- ations by developing parameter-shared twin-gate mechanism. Lei et al. (2018) introduce the simple recurrent unit (SRU), which improves model par- allelization by moving matrix computations out- side the recurrence. Nevertheless, both ATR and SRU perform affine transformations of the previ- ous hidden state for gates, though SRU employs a vector parameter rather than a matrix parame- ter. In addition, SRU heavily relies on its highway component, without which the recurrent compo- nent itself suffers from weak capacity and gener- alization (Lei et al., 2018). In this paper, we propose a lightweight recurrent network (LRN), which combines the strengths of ATR and SRU. The structure of LRN is simple: an input gate and a forget gate are applied to weight the current input and previous hidden state, re- spectively. …

Content truncated to 500 words