Neural Semantic Encoders

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 397–407, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Neural Semantic Encoders Tsendsuren Munkhdalai and Hong Yu University of Massachusetts, MA, USA tsendsuren.munkhdalai@umassmed.edu, hong.yu@umassmed.edu Abstract We present a memory augmented neural network for natural language understand- ing: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access1 multiple and shared mem- ories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question an- swering, sentence classification, document sentiment analysis and machine transla- tion where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encour- aging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU. 1 Introduction Recurrent neural networks (RNNs) have been suc- cessful for modeling sequences (Elman, 1990). Particularly, RNNs equipped with internal short memories, such as long short-term memories (LSTM) (Hochreiter and Schmidhuber, 1997) have achieved a notable success in sequential tasks (Cho et al., 2014; Vinyals et al., 2015). LSTM is powerful because it learns to control its short term memories. However, the short term memories in LSTM are a part of the training parameters. This imposes some practical difficulties in training and modeling long sequences with LSTM. 1By access we mean changing the memory states by the read, compose and write operations. Recently several studies have explored ways of extending the neural networks with an external memory (Graves et al., 2014; Weston et al., 2015; Grefenstette et al., 2015). Unlike LSTM, the short term memories and the training parameters of such a neural network are no longer coupled and can be adapted. In this paper we propose a novel class of memory augmented neural networks called Neu- ral Semantic Encoders (NSE) for natural language understanding. NSE offers several desirable prop- erties. NSE has a variable sized encoding memory which allows the model to access entire input se- quence during the reading process; therefore ef- ficiently delivering long-term dependencies over time. The encoding memory evolves over time and maintains the memory of the input sequence through read, compose and write operations. NSE sequentially processes the input and supports word compositionality inheriting both temporal and hi- erarchical nature of human language. NSE can read from and write to a set of relevant encod- ing memories simultaneously or multiple NSEs can access a shared encoding memory effectively supporting knowledge and representation sharing. NSE is flexible, robust and suitable for practical NLU tasks and can be trained easily by any gradi- ent descent optimizer. We evaluate NSE on five different real tasks. For four of them, our models set new state-of- the-art results. Our results suggest that a NN model with the shared memory …

Content truncated to 500 words