Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5925–5932, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5925 QUOREF: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning Pradeep Dasigi♥ Nelson F. Liu♥♣ Ana Marasovi´c♥ Noah A. Smith♥♣ Matt Gardner♥ ♥Allen Institute for Artificial Intelligence ♣Paul G. Allen School of Computer Science & Engineering, University of Washington {pradeepd,anam,mattg}@allenai.org, {nfliu,nasmith}@cs.washington.edu Abstract Machine comprehension of texts longer than a single sentence often requires coreference resolution. However, most current reading comprehension benchmarks do not contain complex coreferential phenomena and hence fail to evaluate the ability of models to re- solve coreference. We present a new crowd- sourced dataset containing more than 24K span-selection questions that require resolv- ing coreference among entities in over 4.7K English paragraphs from Wikipedia. Obtain- ing questions focused on such phenomena is challenging, because it is hard to avoid lexi- cal cues that shortcut complex reasoning. We deal with this issue by using a strong baseline model as an adversary in the crowdsourcing loop, which helps crowdworkers avoid writing questions with exploitable surface cues. We show that state-of-the-art reading comprehen- sion models perform significantly worse than humans on this benchmark—the best model performance is 70.5 F1, while the estimated human performance is 93.4 F1. 1 Introduction Paragraphs and other longer texts typically make multiple references to the same entities. Track- ing these references and resolving coreference is essential for full machine comprehension of these texts. Significant progress has recently been made in reading comprehension research, due to large crowdsourced datasets (Rajpurkar et al., 2016; Ba- jaj et al., 2016; Joshi et al., 2017; Kwiatkowski et al., 2019, inter alia). However, these datasets focus largely on understanding local predicate- argument structure, with very few questions re- quiring long-distance entity tracking. Obtain- ing such questions is hard for two reasons: (1) teaching crowdworkers about coreference is chal- lenging, with even experts disagreeing on its nu- ances (Pradhan et al., 2007; Versley, 2008; Re- Byzantines were avid players of tavli (Byzantine Greek: τάβλη), a game known in English as backgammon, which is still popular in former Byzantine realms, and still known by the name tavli in Greece. Byzantine nobles were devoted to horsemanship, particularly tzykanion, now known as polo. The game came from Sassanid Persia in the early period and a Tzykanisterion (stadium for playing the game) was built by Theodosius II (r. 408–450) inside the Great Palace of Constantinople. Emperor Basil I (r. 867–886) excelled at it; Emperor Alexander (r. 912–913) died from exhaustion while playing, Emperor Alexios I Komnenos (r. 1081–1118) was injured while playing with Tatikios, and John I of Trebizond (r. 1235–1238) died from a fatal injury during a game. Aside from Constantinople and Trebizond, other Byzantine cities also featured tzykanisteria, most notably Sparta, Ephesus, and Athens, an indication of a thriving urban aristocracy. Q1. What is the Byzantine name of the game that Emperor Basil I excelled at? it …

Content truncated to 500 words