An Empirical Analysis of Multiple-Turn Reasoning Strategies in Reading Comprehension Tasks

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 957–966, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP An Empirical Analysis of Multiple-Turn Reasoning Strategies in Reading Comprehension Tasks Yelong Shen†, Xiaodong Liu†, Kevin Duh‡, Jianfeng Gao† † Microsoft Research, Redmond, WA, USA ‡ Johns Hopkins University, Baltimore, MD, USA †{yeshen,xiaodl,jfgao}@microsoft.com ‡kevinduh@cs.jhu.edu Abstract Reading comprehension (RC) is a chal- lenging task that requires synthesis of in- formation across sentences and multiple turns of reasoning. Using a state-of-the-art RC model, we empirically investigate the performance of single-turn and multiple- turn reasoning on the SQuAD and MS MARCO datasets. The RC model is an end-to-end neural network with iterative attention, and uses reinforcement learn- ing to dynamically control the number of turns. We find that multiple-turn reason- ing outperforms single-turn reasoning for all question and answer types; further, we observe that enabling a flexible number of turns generally improves upon a fixed multiple-turn strategy. We achieve results competitive to the state-of-the-art on these two datasets. 1 Introduction There is an old Chinese proverb that says: “Read a hundred times and the meaning will appear.” Several recent reading comprehension (RC) mod- els have embraced this kind of multiple-turn strat- egy; they generate predictions by making multiple passes through the same text and integrating in- termediate information in the process (Hill et al., 2016; Dhingra et al., 2016; Sordoni et al., 2016; Shen et al., 2016). While state-of-the-art results have been achieved by these models, there has yet to be an in-depth analysis of the impact of the multiple-turn strategy to reasoning. This paper at- tempts to fill this gap. We provide empirical results and analysis on two challenging RC datasets: the Stanford Ques- tion Answering Dataset (SQuAD) (Rajpurkar et al., 2016), and the Microsoft Machine Reading Comprehension Dataset (MS MARCO) (Nguyen et al., 2016). Given a question Q, the RC model is to read passages P and produce an answer A, which could be free-form text or one of the possi- ble candidate spans in the passage. The following example from SQuAD illustrates the need for synthesis of information across sen- tences and multiple turns of reasoning: Q: What collection does the V&A Theator & Performance galleries hold? P: The V&A Theator & Performance gal- leries opened in March 2009. ... They hold the UK’s biggest national collection of material about live performance. To infer the answer (the underlined portion of the passage P), the model needs to first perform coref- erence resolution so that it knows “They” refers “V&A Theator”, then extract the subspan in the direct object corresponding to the answer. This process can be modeled by the repeated process- ing of intermediate states and input in a neural net. To perform the analysis, we adopt the ReasoNet model of Shen et al. (2016). This is an end- to-end neural network that uses an iterative at- tention mechanism to simulate multiple-turn rea- soning in RC. It has achieved strong results on cloze-style RC tasks like …

Content truncated to 500 words