Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists

Proceedings of Recent Advances in Natural Language Processing, pages 1052–1059, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_121 1052 Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists Giancarlo D. Salton ∗ UNOESC Campus Chapec´o Chapec´o, Brazil giancarlo.salton@unoesc.edu.br John D. Kelleher ADAPT Research Centre Technological University Dublin Dublin, Ireland john.d.kelleher@dit.ie Abstract Language Models (LMs) are important components in several Natural Language Processing systems. Recurrent Neural Network LMs composed of LSTM units, especially those augmented with an exter- nal memory, have achieved state-of-the- art results. However, these models still struggle to process long sequences which are more likely to contain long-distance dependencies because of information fad- ing and a bias towards more recent in- formation. In this paper we demonstrate an effective mechanism for retrieving in- formation in a memory augmented LSTM LM based on attending to information in memory in proportion to the number of timesteps the LSTM gating mechanism persisted the information. 1 Introduction Language Models (LM) are important components in Natural Language Processing systems, such as Statistical Machine Translation and Speech Recognition (Schwenk et al., 2012). An LM is generally used to compute the likelihood of a se- quence of words appearing in a given language. Recently, Recurrent Neural Networks LMs (RNN- LMs) have became the state-of-the-art approach to LMs (J´ozefowicz et al., 2016). However, RNN- LMs struggle to keep their level of performance as the length of the input increases. A typical RNN-LM propagates a context vector that integrates information about previous inputs to use for the next prediction. Consequently, the information that is captured at the beginning of a sequence containing a long-distance dependency ∗Work done while the author was at the ADAPT Center and the Dublin Institute of Technology. is likely to have faded from the context by the time the model spans that dependency. To address these limitations, several “memory-augmented” RNN- LMs architectures have been developed that at- tempt to retrieve relevant information from its past timesteps (e.g., Tran et al. (2016), Cheng et al. (2016), Daniluk et al. (2017), Merity et al. (2017), Grave et al. (2017) and Salton et al. (2017)) In this paper, we demonstrate that an efficient and effective mechanism for a memory augmented LSTM based LM (LSTM-LM) to retrieve impor- tant information from its history is to construct a representation of the LSTM unit state history that weights information in proportion to the num- ber of timesteps the unit persisted the informa- tion. Using this strategy reinforces the decisions of the LSTM gating mechanism at each timestep regarding what is important in a sequence. Our models achieve competitive results on the Penn Treebank (Marcus et al., 1994) and on the wiki- text2 (Merity et al., 2017). Structure: §2 presents the architecture of LSTMs; §3 discusses the ef- fect of uniformly weighting the hidden states of an LSTM; §4 illustrates persistence of information in an LSTM and describes our memory augmented LSTM-LM; §5 presents experiments and results; §6 contextualizes our findings; and §7 our conclu- sions. 2 Long Short-Therm …

Content truncated to 500 words