Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios

Proceedings of the 29th International Conference on Computational Linguistics, pages 3606–3630 October 12–17, 2022. 3606 Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios Mana Ashida∗ Yahoo Japan Corporation maashida@yahoo-corp.jp Saku Sugawara National Institute of Informatics saku@nii.ac.jp Abstract The possible consequences for the same con- text may vary depending on the situation we refer to. However, current studies in natural language processing do not focus on situated commonsense reasoning under multiple possi- ble scenarios. This study frames this task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. Our resulting dataset, Possible Stories, consists of more than 4.5K questions over 1.3K story texts in English. We discover that even current strong pretrained language models struggle to answer the questions consis- tently, highlighting that the highest accuracy in an unsupervised setting (60.2%) is far behind human accuracy (92.5%). Through a compar- ison with existing datasets, we observe that the questions in our dataset contain minimal annotation artifacts in the answer options. In addition, our dataset includes examples that re- quire counterfactual reasoning, as well as those requiring readers’ reactions and fictional infor- mation, suggesting that our dataset can serve as a challenging testbed for future studies on situated commonsense reasoning. 1 Introduction Commonsense reasoning, inclusive of counterfac- tual, abductive, and monotonic reasoning, is a core element of language understanding. Researchers are interested in whether these abilities can be learned in systems, and several benchmarks have been proposed to investigate machine common- sense reasoning (Huang et al., 2019; Sap et al., 2019; Aggarwal et al., 2021; Saha et al., 2021). Recent pretrained models have shown competitive results (Khashabi et al., 2020; Lourie et al., 2021). Commonsense reasoning has often been framed as a task to infer whether candidate answers are plausible, such as in the multiple-choice format (Talmor et al., 2019; Sakaguchi et al., 2020). The ∗Work done while at Tokyo Metropolitan University. Original Context Cindy was planning to grow a lot of vegetables this year. She planted vegetable seedlings in her garden. Cindy knew there were hungry groundhogs in the area. She put up a short fence around her garden to protect it. Alternative Ending Collection B: All of the ground animals were kept out, but something was still eating her vegetables. C: She put spikes on the fence to avoid groundhogs and it worked. D: No groundhogs climbed over the fence and Cindy had a good harvest in the fall. Question Writing Q1: Which one of the following is most likely to happen after this if there were other hungry animals? à Option B Q2: What would be the most positive outcome for Cindy's crops? à Option D Original Ending A: The groundhogs climbed over the fence and ate her seedlings. Figure 1: Overview of Possible Stories and its cre- ation process. We ask crowdworkers to produce al- ternative endings given a story text, and then write multiple-choice questions that have a single correct an- swer among the original …

Content truncated to 500 words