A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610 June 6–11, 2021. ©2021 Association for Computational Linguistics 4599 A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers Pradeep Dasigi♣ Kyle Lo♣ Iz Beltagy♣ Arman Cohan♣ Noah A. Smith♦♣ Matt Gardner♣ ♣Allen Institute for AI ♦Paul G. Allen School of CSE, University of Washington {pradeepd,kylel,beltagy,armanc,noah,mattg}@allenai.org Abstract Readers of academic research papers often read with the goal of answering specific ques- tions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information- seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present QASPER, a dataset of 5,049 questions over 1,585 Natu- ral Language Processing papers. Each ques- tion is written by an NLP practitioner who read only the title and abstract of the corre- sponding paper, and the question seeks infor- mation present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting ev- idence to answers. We find that existing mod- els that do well on other QA tasks do not per- form well on answering these questions, un- derperforming humans by at least 27 F1 points when answering them from entire papers, moti- vating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate. 1 Introduction Machines built to assist humans who engage with texts to seek information ought to be designed with an awareness of the information need. Abstractly, the human’s need should define the lens through which the system views the text in order to find desired information. Existing information-seeking machine reading datasets (e.g., Kwiatkowski et al., 2019; Clark et al., 2020) have led to significant progress in reading at scale (e.g., Asai et al., 2020; Guu et al., 2020; Liu et al., 2020). However, most of those benchmarks focus on an “open domain” setting where the questions are not anchored in any particular user context. The result is an emphasis Quasar: Datasets for Question Answering by Search and Reading Abstract We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The QUASAR-S dataset consists of 37000 cloze- style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% & 32.1% for Quasar-S and -T respectively. 3 Dataset Construction Each dataset consists of a collection of records with one QA problem per record. For each record, we include some question text, a context document relevant to the question, a set of candidate solutions, …

Content truncated to 500 words