A Corpus for Commonsense Inference in Story Cloze Test

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3500–3508 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3500 A Corpus for Commonsense Inference in the Story Cloze Test Bingsheng Yao, Ethan Joseph, Julian Lioanag, Mei Si Rensselaer Polytechnic Institute {yaob, josepe2, lioanj, sim}@rpi.edu Abstract The Story Cloze Test (SCT) is designed for training and evaluating machine learning algorithms for narrative understanding and inferences. The SOTA models can achieve over 90% accuracy on predicting the last sentence. However, it has been shown that high accuracy can be achieved by merely using surface-level features. We suspect these models may not truly understand the story. Based on the SCT dataset, we constructed a human-labeled and human-verified commonsense knowledge inference dataset. Given the first four sentences of a story, we asked crowd-source workers to choose from four types of narrative inference for deciding the ending sentence and which sentence contributes most to the inference. We accumulated data on 1871 stories, and three human workers labeled each story. Analysis of the intra-category and inter-category agreements show a high level of consensus. We present two new tasks for predicting the narrative inference categories and contributing sentences. Our results show that transformer-based models can reach SOTA performance on the original SCT task using transfer learning but don’t perform well on these new and more challenging tasks. Keywords: Narrative Reading Comprehension, Commonsense Reasoning, Story Cloze Test, Narrative Reasoning 1. Introduction Narrative Reading Comprehension (RC) and common- sense reasoning are both prevalent Natural Language Processing (NLP) tasks. At the center of this intersec- tion, story understanding is particularly challenging for machine learning algorithms and can even be hard for people (Charniak, 1972), (Oatley, 1994). Stories are more than sequences of words. Commonsense knowl- edge and reasoning are often needed to understand a story. The Story Cloze Test (hereinafter, SCT-v1.0), published in 2016, is a popular dataset for training and evaluating machine story reasoning (Mostafazadeh et al., 2016). Each story is composed of five sentences and follows a character through a series of events to an ending event or situation. The first four sentences of each five-sentence story are provided as “context”, and two alternative sen- tences are provided as the ending, labeled “correct” and “wrong”. Each “context” and corresponding “correct” ending make up a complete five-sentence story. The goal is to predict the correct ending of the story. Ta- ble 1 shows two examples from the SCT-v1.0 dataset. Mostafazadeh et al. (2016) leveraged crowd-workers and carefully defined mechanisms to create the “wrong” endings and make sure they are entirely reasonable when read in isolation and not trivial. Table 2 shows related work performance on the SCT- v1.0 test dataset. Mostafazadeh et al. (2016) provided results of several models at the time when SCT-v1.0 was published, the best of which achieved an accuracy of 59%. Latest works with transfer learning can reach accu- racy comparable to human beings’, which is 100%. Sun et al. (2018) achieved 88.3% accuracy by fine-tuning …

Content truncated to 500 words