POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8674 - 8697 December 7-11, 2022 ©2022 Association for Computational Linguistics POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events Sai Vallurupalli1, Sayontan Ghosh2, Katrin Erk3, Niranjan Balasubramanian2, Francis Ferraro1 1 University of Maryland, Baltimore County, 2 Stony Brook University, 3 University of Texas, Austin kolli@umbc.edu, sagghosh@cs.stonybrook.edu, katrin.erk@utexas.edu, niranjan@cs.stonybrook.edu, ferraro@umbc.edu Abstract Knowledge about outcomes is critical for com- plex event understanding but is hard to ac- quire. We show that by pre-identifying a par- ticipant in a complex event, crowdworkers are able to (1) infer the collective impact of salient events that make up the situation, (2) anno- tate the volitional engagement of participants in causing the situation, and (3) ground the outcome of the situation in state changes of the participants. By creating a multi-step in- terface and a careful quality control strategy, we collect a high quality annotated dataset of 8K short newswire narratives and ROCSto- ries with high inter-annotator agreement (0.74- 0.96 weighted Fleiss Kappa). Our dataset, POQue (Participant Outcome Questions), en- ables the exploration and development of mod- els that address multiple aspects of semantic understanding. Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its out- come, and a participant’s influence over the event culmination. 1 Introduction Situations that people experience or describe can be complex, and developing a computational under- standing of these situations is not straightforward. Consider the short narrative from Fig. 1: After a decade as renters, [the Brofmans] were finally able to buy a small house here four years ago. But if the Argentine government yields to [IMF] pressure to rescind emergency legislation meant to protect ordinary families like the Brof- mans, the couple stand to lose their home and the $32,000 they have paid for it so far. Across multiple, interwoven events with multiple participants, this narrative describes part of the pro- cess of losing one’s house. A possible ending (the loss of a home) is suggested, which is the result of a confluence of these events. This ending can be semantically grounded in various changes of state that the participants experience, though note how the use of counterfactual considerations, condi- tional statements (“if the Argentine government...”), and varying levels of certainty over whether events have actually happened (e.g., realis vs. irrealis) contribute to the difficulty in understanding this complex event (Herman, 2002; Ryan, 1991). Knowledge about how event outcomes affect individual participants can help identify salient events in a narrative, fill in implicit missing infor- mation (LoBue and Yates, 2011) and chain events that lead to improved understanding of complex events (Graesser et al., 1994). To infuse AI mod- els with similar knowledge, narrative comprehen- sion research has focused on learning event rela- tionships (Mostafazadeh et al., 2016; Chambers and Jurafsky, 2008; O’Gorman et al., 2016; Caselli and Vossen, 2016), using temporal (Pustejovsky et …

Content truncated to 500 words