State Gradients for RNN Memory Analysis

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 344–346 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 344 State Gradients for RNN Memory Analysis Lyan Verwimp, Hugo Van hamme, Vincent Renkens, Patrick Wambacq ESAT – PSI, KU Leuven Kasteelpark Arenberg 10 3001 Heverlee, Belgium {firstname}.{lastname}@kuleuven.be Abstract We present a framework for analyzing what the state in RNNs remembers from its input embeddings. We compute the gradients of the states with respect to the input embeddings and decompose the gradient matrix with Sin- gular Value Decomposition to analyze which directions in the embedding space are best transferred to the hidden state space, charac- terized by the largest singular values. We ap- ply our approach to LSTM language models and investigate to what extent and for how long certain classes of words are remembered on average for a certain corpus. Additionally, the extent to which a specific property or re- lationship is remembered by the RNN can be tracked by comparing a vector characterizing that property with the direction(s) in embed- ding space that are best preserved in hidden state space. 1 Introduction Recurrent neural networks (RNNs) are the cur- rent state of the art in many speech and language technology applications, but they are often called ‘black-box’ models since it is hard for humans to interpret what exactly the network has learned. We present a framework to investigate what the states of RNNs remember from their input and for how long. We apply our approach to the cur- rent state of the art in language modeling, long short-term memory (Hochreiter and Schmidhuber, 1997) (LSTM) LMs (Sundermeyer et al., 2012), but it can be applied to other types of RNNs too and to other models with continuous word repre- sentations as input. 2 Average memory of the RNN Our framework is inspired by backpropagation, but instead of computing the gradient of the loss, we compute the gradient of the state with respect to the input embedding, the ‘state gradient’, to capture the influence of the input on the state. To examine how long input words are remembered by the RNN, we calculate the gradient with a certain delay – with respect to the input word embedding a few time steps earlier. The gradient matrix ¯Gτ (averaged over all time steps), where τ is a certain delay, is decomposed with Singular Value Decom- position (SVD): ¯Gτ = U Σ VT = σ1 u1 vT 1 +σ2 u2 vT 2 +. . . (1) We can interpret V as directions in the embed- ding space, Σ as the extent to which the directions in the embedding space can be found in the hid- den state space and U as corresponding directions in the hidden state space. Hence, the directions with the largest singular values (SVs) (lowest in- dex) are directions in embedding space that are best remembered by the RNN. In order to investigate how well the RNN re- members on a corpus level, we …

Content truncated to 500 words