TellMeWhy: A Dataset for Answering Why-Questions in Narratives

TellMeWhy: A Dataset for Answering Why-Questions in Narratives Yash Kumar Lal Stony Brook University ylal@cs.stonybrook.edu Nathanael Chambers US Naval Academy nchamber@usna.edu Raymond Mooney University of Texas, Austin mooney@cs.utexas.edu Niranjan Balasubramanian Stony Brook University niranjan@cs.stonybrook.edu Abstract Answering questions about why characters perform certain actions is central to under- standing and reasoning about narratives. De- spite recent progress in QA, it is not clear if existing models have the ability to answer “why” questions that may require common- sense knowledge external to the input narra- tive. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form an- swers concerning why characters in short nar- ratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limita- tions of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on ques- tions whose answers are external to the narra- tive, thus providing a challenge for future QA and narrative understanding research. 1 Introduction The actions people perform are steps of plans to achieve their desired goals. When interpreting lan- guage, humans naturally understand the reasons behind described actions, even when the reasons are left unstated (Schank and Abelson, 1975). For NLP systems, answering questions about why peo- ple perform actions in a narrative can test this abil- ity. Answering such questions often requires filling the implicit gaps in the story itself. Consider this narrative from ROCSto- ries (Mostafazadeh et al., 2016b): Rudy was convinced that bottled waters all tasted the same. He went to the store and bought several popular brands. He went back home and set them all on a table. He spent several hours tasting them one by one. He came to the conclusion that they actually did taste different. Now try to answer the question, “Why did he go to the store and buy several popular brands?” The answer “he wanted to taste test” is not explicit in the narrative and requires us to read between the lines to fill in the gaps (Norvig, 1987). While humans can visualise and process the events in a story to hypothesize why they might have occurred (Kintsch and Dijk, 1978), current NLP systems fall well short of exhibiting similar capabilities. They are unable to adequately formulate the reasons be- hind actions in specific contexts. How can we get NLP models to reason about why actions are performed? One way is to consider theories like script learning (Schank and Abelson, 1975; Pichotta and Mooney, 2014) or learning from co-occurrence (Chambers and Jurafsky, 2009). But they only partially capture this type of knowledge – much like other forms of commonsense knowl- edge, the reasons for why actions are performed are often left implicit in text. Even though there are many large scale QA datasets, they rarely contain questions about …

Content truncated to 500 words