Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications

Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications Rumen Dangovski*,1, Li Jing*,1, Preslav Nakov2, Mi´co Tatalovi´c1,3, Marin Soljaˇci´c1 *equal contribution 1Massachusetts Institute of Technology 2Qatar Computing Research Institute, HBKU 3Association of British Science Writers {rumenrd, ljing}@mit.edu, pnakov@qf.org.qa, {mico, soljacic}@mit.edu Abstract Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has be- come a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conven- tional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotatio- nal Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and mem- ory recall much better than LSTMs/GRUs. We further demonstrate that by replacing LSTM/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art. 1 Introduction An important element of the ongoing neural revo- lution in Natural Language Processing (NLP) is the rise of Recurrent Neural Networks (RNNs), which have become a standard tool for addressing a num- ber of tasks ranging from language modeling, part-of-speech tagging and named entity recogni- tion to neural machine translation, text summa- rization, question answering, and building chatbots/ dialog systems. As standard RNNs suffer from exploding/ vanishing gradient problems, alternatives such as long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) or gated recurrent units (GRUs) (Cho et al., 2014) have been proposed and have now become standard. Nevertheless, LSTMs and GRUs fail to demon- strate really long-term memory capabilities or efficient recall on synthetic tasks (see Figure 1). Figure 1 shows that when RNN units are fed a long string (e.g., emojis in Figure 1(a)), they struggle to represent the input in their memory, which results in recall or copy mistakes. The origins of these issues are two-fold: (i) a single hidden state cannot memorize complicated sequential dynamics and (ii) the hidden state is not manipulated well, resulting in information loss. Typically, these are addressed separately: by using external memory for (i), and gated mechanisms for (ii). Here, we solve (i) and (ii) jointly by proposing a novel RNN unit, Rotational Unit of Memory (RUM), that manipulates the hidden state by rotating it in an Euclidean space, resulting in a better information flow. This remedy to (ii) affects (i) to the extent that the external memory is less needed. As a proof of concept, in Figure 1(a), RUM recalls correctly a faraway emoji. We further show that RUM is fit for real-world NLP tasks. In Figure 1(b), a RUM-based seq2seq model produces a better summary than what a standard LSTM-based seq2seq model yields. In this particular example, LSTM falls into the …

Content truncated to 500 words