A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

Proceedings of NAACL-HLT 2016, pages 839–849, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories Nasrin Mostafazadeh1, Nathanael Chambers2, Xiaodong He3, Devi Parikh4, Dhruv Batra4, Lucy Vanderwende3, Pushmeet Kohli3, James Allen1,5 1 University of Rochester, 2 United States Naval Academy, 3 Microsoft Research, 4 Virginia Tech, 5 The Institute for Human & Machine Cognition {nasrinm,james}@cs.rochester.edu, nchamber@usna.edu, {parikh,dbatra}@vt.edu, {xiaohe,lucyv,pkohli}@microsoft.com Abstract Representation and learning of commonsense knowledge is one of the foundational prob- lems in the quest to enable deep language un- derstanding. This issue is particularly chal- lenging for understanding casual and corre- lational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the ‘Story Cloze Test’. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of 50k five-sentence commonsense stories, ROCSto- ries, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high qual- ity collection of everyday life stories that can also be used for story generation. Experimen- tal evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learn- ing, and offer suggestions for deeper language understanding. 1 Introduction Story understanding is an extremely challenging task in natural language understanding with a long- running history in AI (Charniak, 1972; Winograd, 1972; Turner, 1994; Schubert and Hwang, 2000). Recently, there has been a renewed interest in story and narrative understanding based on progress made in core NLP tasks. This ranges from generic story telling models to building systems which can com- pose meaningful stories in collaboration with hu- mans (Swanson and Gordon, 2008). Perhaps the biggest challenge of story understanding (and story generation) is having commonsense knowledge for the interpretation of narrative events. The question is how to provide commonsense knowledge regard- ing daily events to machines. A large body of work in story understanding has focused on learning scripts (Schank and Abel- son, 1977). Scripts represent structured knowledge about stereotypical event sequences together with their participants. It is evident that various NLP applications (text summarization, co-reference res- olution, question answering, etc.) can hugely ben- efit from the rich inferential capabilities that struc- tured knowledge about events can provide. Given that developing hand-built scripts is extremely time- consuming, there is a serious need for automati- cally induced scripts. Most relevant to this issue is work on unsupervised learning of ‘narrative chains’ (Chambers and Jurafsky, 2008) and event schemas (Chambers and Jurafsky, 2009; Balasubramanian et al., 2013; Cheung et al., 2013; Nguyen et al., 2015). …

Content truncated to 500 words