Long Short-Term Memory-Networks for Machine Reading

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 551–561, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Long Short-Term Memory-Networks for Machine Reading Jianpeng Cheng, Li Dong and Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB {jianpeng.cheng,li.dong}@ed.ac.uk, mlap@inf.ed.ac.uk Abstract In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a ma- chine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and atten- tion. The reader extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell. This en- ables adaptive memory usage during recur- rence with neural attention, offering a way to weakly induce relations among tokens. The system is initially designed to process a single sequence but we also demonstrate how to inte- grate it with an encoder-decoder architecture. Experiments on language modeling, sentiment analysis, and natural language inference show that our model matches or outperforms the state of the art. 1 Introduction How can a sequence-level network induce relations which are presumed latent during text processing? How can a recurrent network attentively memorize longer sequences in a way that humans do? In this paper we design a machine reader that automatically learns to understand text. The term machine read- ing is related to a wide range of tasks from answer- ing reading comprehension questions (Clark et al., 2013), to fact and relation extraction (Etzioni et al., 2011; Fader et al., 2011), ontology learning (Poon and Domingos, 2010), and textual entailment (Da- gan et al., 2005). Rather than focusing on a specific task, we develop a general-purpose reading simula- tor, drawing inspiration from human language pro- cessing and the fact language comprehension is in- cremental with readers continuously extracting the meaning of utterances on a word-by-word basis. In order to understand texts, our machine reader should provide facilities for extracting and repre- senting meaning from natural language text, storing meanings internally, and working with stored mean- ings to derive further consequences. Ideally, such a system should be robust, open-domain, and de- grade gracefully in the presence of semantic rep- resentations which may be incomplete, inaccurate, or incomprehensible. It would also be desirable to simulate the behavior of English speakers who pro- cess text sequentially, from left to right, fixating nearly every word while they read (Rayner, 1998) and creating partial representations for sentence pre- fixes (Konieczny, 2000; Tanenhaus et al., 1995). Language modeling tools such as recurrent neural networks (RNN) bode well with human reading be- havior (Frank and Bod, 2011). RNNs treat each sen- tence as a sequence of words and recursively com- pose each word with its previous memory, until the meaning of the whole sentence has been derived. In practice, however, sequence-level networks are met with at least three challenges. The first one concerns model training problems associated with vanishing and exploding gradients (Hochreiter, 1991; Bengio et al., 1994), …

Content truncated to 500 words