TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1158–1172, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1158 TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions Qiang Ning♦ Hao Wu♣ Rujun Han♠ Nanyun Peng♠ Matt Gardner♦ Dan Roth♥ ♦Allen Institute for AI ♠University of Southern California♣Hooray Data Co., Ltd ♥University of Pennsylvania {qiangn,mattg}@allenai.org, haowu@hooray.ai {rujunhan,npeng}@isi.edu, danroth@seas.upenn.edu Abstract A critical part of reading is being able to un- derstand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading comprehension benchmarks have practically no questions that test temporal phenomena, so systems trained on these benchmarks have no capacity to answer questions such as “what happened before/after [some event]?” We introduce TORQUE, a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships. Results show that RoBERTa-large achieves an exact-match score of 51% on the test set of TORQUE, about 30% behind human performance.1 1 Introduction Time is important for understanding events and stories described in natural language text such as news articles, social media, financial reports, and electronic health records (Verhagen et al., 2007, 2010; UzZaman et al., 2013; Minard et al., 2015; Bethard et al., 2016, 2017; Laparra et al., 2018). For instance, “he won the championship yester- day” is different from “he will win the champi- onship tomorrow”: he may be celebrating if he has already won it, while if he has not, he is probably still preparing for the game tomorrow. The exact time of an event is often implicit in text. For instance, if we read that a woman is “ex- pecting the birth of her first child”, we know that the birth is in the future, while if she is “mourning the death of her mother”, the death is in the past. These relationships between an event and a time point (e.g., “won the championship yesterday”) or between two events (e.g., “expecting” is before 1https://allennlp.org/torque.html Heavy snow is causingdisruption to transport across the UK, with heavy rainfall bringingfloodingto the south-west of England. Rescuers searchingfor a woman trappedin a landslide at her home in Looe, Cornwall, said they had founda body. Q1: What events have already finished? A: searching trapped landslide said found Q2: What events have begun but has not finished? A: snow causing disruption rainfall bringing flooding Q3: What will happen in the future? A: No answers. Q4: What happened before a woman was trapped? A: landslide Q5: What had started before a woman was trapped? A: snow rainfall landslide Q6: What happened while a woman was trapped? A: searching Q7: What happened after a woman was trapped? A: searching said found Q8: What happened at about the same time as the snow? A: rainfall Q9: What happened after the snow started? A: causing disruption bringing flooding searching trapped landslide said found Q10: What happened before the snow started? A: No answers. warm-up User-provided User-provided Figure 1: Example annotation …

Content truncated to 500 words