MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, Manfred Pinkal Saarland University Saarbr¨ucken, Germany {simono|ashutosh|mroth|stth|pinkal}@coli.uni-saarland.de Abstract We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community. Keywords: machine comprehension, reading comprehension, commonsense knowledge, script knowledge 1. Introduction Ambiguity and implicitness are inherent properties of natu- ral language that cause challenges for computational mod- els of language understanding. In everyday communica- tion, people assume a shared common ground which forms a basis for efficiently resolving ambiguities and for infer- ring implicit information. Thus, recoverable information is often left unmentioned or underspecified. Such information may include encyclopedic and commonsense knowledge. This work focuses on commonsense knowledge about ev- eryday activities, so-called scripts. This paper introduces a dataset to evaluate natural lan- guage understanding approaches with a focus on interpreta- tion processes requiring inference based on commonsense knowledge. In particular, we present MCScript, a dataset for assessing the contribution of script knowledge to ma- chine comprehension. Scripts are sequences of events de- scribing stereotypical human activities (also called scenar- ios), for example baking a cake or taking a bus (Schank and Abelson, 1975). To illustrate the importance of script knowledge, consider Example (1): (1) The waitress brought Rachel’s order. She ate the food with great pleasure. Without using commonsense knowledge, it may be diffi- cult to tell who ate the food: Rachel or the waitress. In contrast, if we utilize commonsense knowledge, in particu- lar, script knowledge about the EATING IN A RESTAURANT scenario, we can make the following inferences: Rachel is most likely a customer, since she received an order. It is usually the customer, and not the waitress, who eats the or- dered food. So She most likely refers to Rachel. Various approaches for script knowledge extraction and processing have been proposed in recent years. However, systems have been evaluated for specific aspects of script knowledge only, such as event ordering (Modi and Titov, 2014a; Modi and Titov, 2014b), event paraphrasing (Reg- neri et al., 2010; Wanzare et al., 2017) or event prediction (namely, the narrative cloze task (Chambers and Jurafsky, T I wanted to plant a tree. I went to the home and garden store and picked a nice oak. After- wards, I planted it in my garden. Q1 What was used to dig the hole? a. a shovel b. his bare …

Content truncated to 500 words