IIRC: A Dataset of Incomplete Information Reading Comprehension Questions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1137–1147, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1137 IIRC: A Dataset of Incomplete Information Reading Comprehension Questions James Ferguson♦∗ Matt Gardner♣ Hannaneh Hajishirzi♦♣ Tushar Khot♣ Pradeep Dasigi♣ ♦University of Washington ♣Allen Institute for AI {jfferg,hannaneh}@cs.washington.edu {mattg,tushark,pradeepd}@allenai.org Abstract Humans often have to read multiple doc- uments to address their information needs. However, most existing reading comprehen- sion (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a poten- tial lack of sufficient information and locat- ing sources for that information. To fill this gap, we present a dataset, IIRC, with more than 13K questions over paragraphs from En- glish Wikipedia that provide only partial in- formation to answer them, with the missing information occurring in one or more linked documents. The questions were written by crowd workers who did not have access to any of the linked documents, leading to questions that have little lexical overlap with the con- texts where the answers appear. This process also gave many questions without answers, and those that require discrete reasoning, in- creasing the difficulty of the task. We fol- low recent modeling work on various reading comprehension datasets to construct a baseline model for this dataset, finding that it achieves 31.1% F1 on this task, while estimated human performance is 88.4%. The dataset, code for the baseline system, and a leaderboard can be found at https://allennlp.org/iirc. 1 Introduction Humans often read text with the goal of obtain- ing information. Given that a single document is unlikely to contain all the information a reader might need, the reading process frequently involves identifying the information present in the given document, and what is missing, followed by locat- ing a different source that could potentially con- tain the missing information. Most recent read- ∗Work done as an intern at the Allen Institute for AI. Figure 1: An example from IIRC. At the top is a con- text paragraph which provides only partial information required to answer the question. The bold spans in the context indicate links to other Wikipedia pages. The colored boxes below the question show snippets from four of these pages that provide the missing informa- tion for answering the question. The answer is the un- derlined span. ing comprehension tasks, such as SQuAD 2.0 (Ra- jpurkar et al., 2018), DROP (Dua et al., 2019b), or Quoref (Dasigi et al., 2019), evaluate models using a relatively simpler setup where all the information required to answer the questions (including judg- ing them as being unanswerable) is provided in the associated contexts. While this setup has led to sig- nificant advances in reading comprehension (Ran et al., 2019; Zhang et al., 2020), the tasks are still limited since they do not evaluate the capability of models at identifying precisely what information, if any, is missing to answer a question, and where that information might …

Content truncated to 500 words