Pointing to Select: A Fast Pointer-LSTM for Long Text Classification

Proceedings of the 28th International Conference on Computational Linguistics, pages 6184–6193 Barcelona, Spain (Online), December 8-13, 2020 6184 Pointing to Select: A Fast Pointer-LSTM for Long Text Classification Jinhua Du, Yan Huang and Karo Moilanen Investments AI, AIG (American International Group, Inc.) {jinhua.du, yan.huang, karo.moilanen}@aig.com Abstract Recurrent neural networks (RNNs) suffer from well-known limitations and complications which include slow inference and vanishing gradients when processing long sequences in text classi- fication. Recent studies have attempted to accelerate RNNs via various ad hoc mechanisms to skip irrelevant words in the input. However, word skipping approaches proposed to date effec- tively stop at each or a given time step to decide whether or not a given input word should be skipped, breaking the coherence of input processing in RNNs. Furthermore, current methods cannot change skip rates during inference and are consequently unable to support different skip rates in demanding real-world conditions. To overcome these limitations, we propose Pointer- LSTM, a novel LSTM framework which relies on a pointer network to select important words for target prediction. The model maintains a coherent input process for the LSTM modules and makes it possible to change the skip rate during inference. Our evaluation on four public data sets demonstrates that Pointer-LSTM (a) is 1.1x∼3.5x faster than the standard LSTM architecture; (b) is more accurate than Leap-LSTM (the state-of-the-art LSTM skipping model) at high skip rates; and (c) reaches robust accuracy levels even when the skip rate is changed during inference. 1 Introduction Recurrent neural networks (RNNs), such as Long Short-term Memory (LSTM) (Hochreiter and Schmid- huber, 1997) and Gated Recurrent Unit (GRU) (Chung et al., 2014), are popular architectural choices for many sequential text processing tasks. Typically, RNNs read in either all or a fixed-length slice of tokens sequentially, and output a distributed representation for each token. While this paradigm is essential for some applications such as machine translation (Bahdanau et al., 2015), it may result in suboptimal or practically prohibitive processing times for long input sequences which are not uncommon in document classification (Liu et al., 2015) and complex question answering (Tan et al., 2015). Building on wider, es- tablished observations that (i) not all input tokens are equally important in many tasks (Seo et al., 2017); (ii) text can be comprehended even through partial reading due to linguistic redundancy effects (Yu et al., 2017); and (iii) attention, fixations, and saccades govern human reading in general (Hahn and Keller, 2016), recent RNN optimisation efforts have focused on mechanisms to skip irrelevant, unimportant, or redundant words in the input (Yu et al., 2017; Campos et al., 2017; Seo et al., 2017; Liu et al., 2018; Hansen et al., 2019; Huang et al., 2019). We refer to such models as selective RNNs. In general, existing selective RNNs use a decision-making component to predict whether or not a word should be skipped (skimmed). In a typical selective RNN workflow, the RNN’s input steps are interrupted by the decision network to make a decision to copy the previous …

Content truncated to 500 words