Ruminating Reader: Reasoning with Gated Multi-hop Attention

Proceedings of the Workshop on Machine Reading for Question Answering, pages 1–11 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 1 Ruminating Reader: Reasoning with Gated Multi-Hop Attention Yichen Gong and Samuel R. Bowman New York University New York, NY {yichen.gong,bowman}@nyu.edu Abstract To answer the question in machine com- prehension (MC) task, the models need to establish the interaction between the question and the context. To tackle the problem that the single-pass model can- not reflect on and correct its answer, we present Ruminating Reader. Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BIDAF). We propose novel layer struc- tures that construct a query aware con- text vector representation and fuse encod- ing representation with intermediate rep- resentation on top of BIDAF model. We show that a multi-hop attention mecha- nism can be applied to a bi-directional attention structure. In experiments on SQuAD, we find that the Reader outper- forms the BIDAF baseline by 2.1 F1 score and 2.7 EM score. Our analysis shows that different hops of the attention have differ- ent responsibilities in selecting answers. 1 Introduction The majority of recorded human knowledge is cir- culated in unstructured natural language. It is tremendously valuable to allow machines to read and comprehend the text knowledge. Machine comprehension (MC)—especially in the form of question answering (QA)—is therefore attracting a significant amount of attention from the ma- chine learning community. Recently introduced large-scale datasets like CNN/Daily Mail (Her- mann et al., 2015), the Stanford Question Answer- ing Dataset (SQuAD; Rajpurkar et al., 2016) and the Microsoft MAchine Reading COmprehension Dataset (MS-MARCO; Nguyen et al., 2016) have (a) The high-level structure of BIDAF. (b) The high-level structure of Ruminating Reader. Figure 1: The high-level comparison between BIDAF and Ruminating Reader. allow data-driven methods, including deep learn- ing, to become viable. Recent approaches toward solving machine comprehension tasks using neural networks can be viewed as falling into two broad categories: single-pass reasoners and multiple-pass reasoners. Single-pass models read a question and a source text once and often adopt the differentiable atten- tion mechanism that emphasizes important parts of the context related to the question. BIDAF (Seo et al., 2017) represents one of the 2 competitive single-pass models in Machine Com- prehension. BIDAF uses a bi-directional attention matrix which calculates the correlations between each word pair in context and query to build query- aware context representation. However, BIDAF and some similar models miss some questions be- cause they don’t have the capacity to reflect on problematic candidate answers and revise their de- cisions. When humans are reading a text with the goal of answering a question, they tend to read it multiple times to get a better understanding of the context and question, and to give a better response. With this intuition, recent multi-pass models revisit the question and the context passage (or ruminate) to infer the relations between the context, the ques- tion and the answer. We propose an …

Content truncated to 500 words