A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension

Proceedings of the Workshop on Machine Reading for Question Answering, pages 21–30 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 21 A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension Seunghak Yu Sathish Indurthi Seohyun Back Haejun Lee Samsung Research, Seoul, Korea {seunghak.yu, s.indurthi, scv.back}@samsung.com Abstract Reading Comprehension (RC) of text is one of the fundamental tasks in natu- ral language processing. In recent years, several end-to-end neural network mod- els have been proposed to solve RC tasks. However, most of these models suffer in reasoning over long documents. In this work, we propose a novel Memory Aug- mented Machine Comprehension Network (MAMCN) to address long-range depen- dencies present in machine reading com- prehension. We perform extensive exper- iments to evaluate proposed method with the renowned benchmark datasets such as SQuAD, QUASAR-T, and TriviaQA. We achieve the state of the art performance on both the document-level (QUASAR-T, TriviaQA) and paragraph-level (SQuAD) datasets compared to all the previously published approaches. 1 Introduction Reading Comprehension (RC) is essential for understanding human knowledge written in text form. One possible way of measuring RC is by formulating it as answer span prediction style Question Answering (QA) task, which is finding an answer to the question based on the given doc- ument(s). Recently, influential deep learning ap- proaches have been proposed to solve this QA task. Wang and Jiang (2017); Seo et al. (2017) pro- pose the attention mechanism between question and context for question-aware contextual repre- sentation. Wang et al. (2017) refine these con- textual representations by using self-attention to improve the performance. Even further perfor- mance improvement is gained by using contextu- alized word representations for query and context (Salant and Berant, 2017; Peters et al., 2018; Yu et al., 2018). Based on those approaches, several methods have successfully made progress towards reaching human-level performance on SQuAD (Rajpurkar et al., 2016). Each training example in the SQuAD only has the relevant paragraph with the corre- sponding answer. However, most of the docu- ments present in the real-world are long, contain- ing relevant and irrelevant paragraphs, and do not guarantee answer presence. Therefore the models proposed to solve SQuAD have difficulty in apply- ing to real-world documents (Joshi et al., 2017). Recently, QUASAR-T (Dhingra et al., 2017) and TriviaQA (Joshi et al., 2017) datasets have been proposed to resemble real-world document. These datasets use document-level evidence as training example instead of using only the relevant para- graph and evidence does not guarantee answer presence, which makes them more realistic. To effectively comprehend long documents present in the QUASAR-T and TriviaQA datasets, the QA models have to resolve long-range de- pendencies present in these documents. In this work, we build a QA model that can understand long documents by utilizing Memory Augmented Neural Networks (MANNs) (Graves et al., 2014; Weston et al., 2015b). This type of neural net- works decouples the memory capacity from the number of model parameters. While there have been several attempts to use MANNs in managing long-range dependencies, …

Content truncated to 500 words