Hierarchical Memory Networks for Answer Selection on Unknown Words

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2290–2299, Osaka, Japan, December 11-17 2016. Hierarchical Memory Networks for Answer Selection on Unknown Words Jiaming Xua,∗, Jing Shia,∗, Yiqun Yaoa, Suncong Zhenga, Bo Xua, Bo Xua,b aInstitute of Automation, Chinese Academy of Sciences (CAS). Beijing, China bCenter for Excellence in Brain Science and Intelligence Technology, CAS. China {jiaming.xu, shijing2014, yaoyiqun2014}@ia.ac.cn {suncong.zheng, boxu, xubo}@ia.ac.cn Abstract Recently, end-to-end memory networks have shown promising results on Question Answering task, which encode the past facts into an explicit memory and perform reasoning ability by mak- ing multiple computational steps on the memory. However, memory networks conduct the rea- soning on sentence-level memory to output coarse semantic vectors and do not further take any attention mechanism to focus on words, which may lead to the model lose some detail informa- tion, especially when the answers are rare or unknown words. In this paper, we propose a novel Hierarchical Memory Networks, dubbed HMN. First, we encode the past facts into sentence- level memory and word-level memory respectively. Then, k-max pooling is exploited following reasoning module on the sentence-level memory to sample the k most relevant sentences to a question and feed these sentences into attention mechanism on the word-level memory to focus the words in the selected sentences. Finally, the prediction is jointly learned over the outputs of the sentence-level reasoning module and the word-level attention mechanism. The experimental results demonstrate that our approach successfully conducts answer selection on unknown words and achieves a better performance than memory networks. 1 Introduction With the recent resurgence of interest in Deep Neural Networks (DNN), many researchers have concen- trated on using deep learning to solve natural language processing (NLP) tasks (Collobert et al., 2011; Sutskever et al., 2014; Zeng et al., 2014; Feng et al., 2015). The main merits of these representation learning based methods are that they do not rely on any linguistic tools and can be applied to differ- ent languages or domains. However, the memory of these methods, such as Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014) com- pressing all the external sentences into a fixed-length vector, is typically too small to accurately remember facts from the past, and may lose important details for response generation (Shang et al., 2015). Due to the drawback, these traditional DNN models encounter great limitation on Question Answering (QA), as a complex NLP task, which requires deep understanding of semantic abstraction and reasoning over facts that are relevant to a question (Hermann et al., 2015; Yu et al., 2015). Recently, lots of deep learning methods with explicit memory and attention mechanism are explored for Question Answering (QA) task, such as Memory Networks (MemNN) (Sukhbaatar et al., 2015), Neural Machine Translation (NMT) and Neural Turing Machine (NTM) (Yu et al., 2015). These meth- ods exploit a external memory to store the past sentences with a continuous representation and utilize attention mechanism to automatically soft-search for …

Content truncated to 500 words