Interactive Machine Comprehension with Information Seeking Agents

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2325–2338 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2325 Interactive Machine Comprehension with Information Seeking Agents Xingdi Yuan†∗ Jie Fu‡♠∗ Marc-Alexandre Cˆot´e† Yi Tay♦ Christopher Pal‡♠♥ Adam Trischler† †Microsoft Research, Montr´eal ‡Polytechnique Montr´eal ♠ Mila ♦ Nanyang Technological University ♥ Canada CIFAR AI Chair eric.yuan@microsoft.com jie.fu@polymtl.ca Abstract Existing machine reading comprehension (MRC) models do not scale effectively to real- world applications like web-level information retrieval and question answering (QA). We ar- gue that this stems from the nature of MRC datasets: most of these are static environments wherein the supporting documents and all nec- essary information are fully observed. In this paper, we propose a simple method that re- frames existing MRC datasets as interactive, partially observable environments. Specifi- cally, we “occlude” the majority of a doc- ument’s text and add context-sensitive com- mands that reveal “glimpses” of the hidden text to a model. We repurpose SQuAD and NewsQA as an initial case study, and then show how the interactive corpora can be used to train a model that seeks relevant informa- tion through sequential decision making. We believe that this setting can contribute in scal- ing models to web-level QA scenarios.1 1 Introduction Many machine reading comprehension (MRC) datasets have been released in recent years (Ra- jpurkar et al., 2016; Trischler et al., 2016; Nguyen et al., 2016; Reddy et al., 2018; Yang et al., 2018) to benchmark a system’s ability to understand and rea- son over natural language. Typically, these datasets require an MRC model to read through a document to answer a question about information contained therein. The supporting document is, more often than not, static and fully observable. This raises concerns, since models may find answers simply through shallow pattern matching; e.g., syntactic similarity between the words in questions and documents. As ∗ Equal contribution. 1The dataset and implementation of our baseline agents are publicly available at https://github.com/ xingdi-eric-yuan/imrc_public. Question: What was the Harvard endowment total in 2011 ? o1 Harvard has the largest university endowment in the world . WWWWWWWWWWWWWWWWWWWWwwwwwnext a1 o2 At the end of June 2009, it was worth $25.7 billion, about 30% less than at the same time in 2008. Ctrl+F Harvard a2 o3 In December 2008, Harvard announced that its endowment had lost 22% from July to October 2008, necessitating budget cuts. Ctrl+F 2011 a3 o4 As of September 2011 , it had nearly regained the loss suffered during the 2008 recession . Ctrl+F 2011 a4 o5 It was worth $ 32 billion in 2011 , up from $ 28 billion in September 2010 and $ 26 billion in 2009 . stop a5 Prediction: $ 32 billion Table 1: Example of the interactive machine reading comprehension behavior. pointed out by Sugawara et al. (2018), for ques- tions starting with when, models tend to predict the only date/time answer in the supporting document. Such behavior limits the generality and usefulness of MRC models, and suggests that …

Content truncated to 500 words