ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7543–7559 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7543 ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations Rujun Han1 I-Hung Hsu1 Jiao Sun1 Julia Baylon2 Qiang Ning3∗ Dan Roth4 Nanyun Peng1,2 1University of Southern California 2University of California, Los Angeles 3Amazon 4University of Pennsylvania {rujunhan,ihunghsu,jiaosun}@usc.edu; juliabaylon@ucla.edu qning@amazon.com; danroth@seas.upenn.edu violetpeng@cs.ucla.edu Abstract Understanding how events are semantically re- lated to each other is the essence of read- ing comprehension. Recent event-centric read- ing comprehension datasets focus mostly on event arguments or temporal relations. While these tasks partially evaluate machines’ ability of narrative understanding, human-like read- ing comprehension requires the capability to process event-based information beyond argu- ments and temporal reasoning. For example, to understand causality between events, we need to infer motivation or purpose; to estab- lish event hierarchy, we need to understand the composition of events. To facilitate these tasks, we introduce ESTER, a comprehensive ma- chine reading comprehension (MRC) dataset for Event Semantic Relation Reasoning. The dataset leverages natural language queries to reason about the five most common event semantic relations, provides more than 6K questions, and captures 10.1K event relation pairs. Experimental results show that the cur- rent SOTA systems achieve 22.1%, 63.3% and 83.5% for token-based exact-match (EM), F1 and event-based HIT@1 scores, which are all significantly below human performances (36.0%, 79.6%, 100% respectively), highlight- ing our dataset as a challenging benchmark. 1 1 Introduction Narratives such as stories and news articles are composed of series of events (Carey and Snod- grass, 1999; Harmon, 2012). Understanding how events are logically connected is essential for read- ing comprehension (Caselli and Vossen, 2017; Mostafazadeh et al., 2016b). For example, Fig- ure 1 illustrates several pairwise relations for events in the given passage: “the deal” can be consid- ered as the same event of “Paramount purchased ∗ Part of the work was done while the author was at the Allen Institute for AI. 1Data, models and reproduction code are available here: https://github.com/PlusLabNLP/ESTER. Paramount film studios has purchased DreamWorks for 1.6 billion dollars in cash, the company said in a statement Sunday. DreamWorks, which created successful computer animated features such as Shrek, was also sought after by NBC Universal. In the deal, Paramount assumed DreamWorks debt. The final deal gives Paramount access to 59 films in DreamWorks' library. Paramount takes over all of DreamWorks' current projects. deal created sought after assumed access takes over purchased Co-refers Conditional Sub-events Figure 1: A graph illustration of event semantic rela- tions in narratives. We use trigger words to represent events in this graph. DreamWorks,” forming a coreference relation; it is also a complex event that contains “assumed debt,” “gives access” and “takes over projects” as its sub- events. The event “sought after” is facilitated by a previous event “created features.” By capturing these event semantic relations, people can often grasp the gist of a story. Therefore, for machines to achieve human-level narrative understanding, we need to test …

Content truncated to 500 words