Recurrent Memory Networks for Language Modeling

Proceedings of NAACL-HLT 2016, pages 321–331, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Recurrent Memory Networks for Language Modeling Ke Tran Arianna Bisazza Christof Monz Informatics Institute, University of Amsterdam Science Park 904, 1098 XH Amsterdam, The Netherlands {m.k.tran,a.bisazza,c.monz}@uva.nl Abstract Recurrent Neural Networks (RNNs) have ob- tained excellent result in many natural lan- guage processing (NLP) tasks. However, un- derstanding and interpreting the source of this success remains a challenge. In this paper, we propose Recurrent Memory Network (RMN), a novel RNN architecture, that not only am- plifies the power of RNN but also facilitates our understanding of its internal functioning and allows us to discover underlying patterns in data. We demonstrate the power of RMN on language modeling and sentence comple- tion tasks. On language modeling, RMN out- performs Long Short-Term Memory (LSTM) network on three large German, Italian, and English dataset. Additionally we perform in- depth analysis of various linguistic dimen- sions that RMN captures. On Sentence Com- pletion Challenge, for which it is essential to capture sentence coherence, our RMN obtains 69.2% accuracy, surpassing the previous state of the art by a large margin.1 1 Introduction Recurrent Neural Networks (RNNs) (Elman, 1990; Mikolov et al., 2010) are remarkably powerful mod- els for sequential data. Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), a spe- cific architecture of RNN, has a track record of suc- cess in many natural language processing tasks such as language modeling (J´ozefowicz et al., 2015), de- pendency parsing (Dyer et al., 2015), sentence com- 1Our code and data are available at https://github. com/ketranm/RMN pression (Filippova et al., 2015), and machine trans- lation (Sutskever et al., 2014). Within the context of natural language process- ing, a common assumption is that LSTMs are able to capture certain linguistic phenomena. Evidence sup- porting this assumption mainly comes from evaluat- ing LSTMs in downstream applications: Bowman et al. (2015) carefully design two artificial datasets where sentences have explicit recursive structures. They show empirically that while processing the in- put linearly, LSTMs can implicitly exploit recursive structures of languages. Filippova et al. (2015) find that using explicit syntactic features within LSTMs in their sentence compression model hurts the per- formance of overall system. They then hypothesize that a basic LSTM is powerful enough to capture syntactic aspects which are useful for compression. To understand and explain which linguistic di- mensions are captured by an LSTM is non-trivial. This is due to the fact that the sequences of input histories are compressed into several dense vectors by the LSTM’s components whose purposes with re- spect to representing linguistic information is not ev- ident. To our knowledge, the only attempt to better understand the reasons of an LSTM’s performance and limitations is the work of Karpathy et al. (2015) by means of visualization experiments and cell acti- vation statistics in the context of character-level lan- guage modeling. Our work is motivated by the difficulty in un- derstanding and interpreting existing RNN architec- tures from a linguistic …

Content truncated to 500 words