MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 103–117 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 103 MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants Simon Ostermann1 Michael Roth1,2 Manfred Pinkal1 1Saarland University 2Stuttgart University {simono|pinkal}@coli.uni-saarland.de rothml@ims.uni-stuttgart.de Abstract We introduce MCScript2.0, a machine comprehension corpus for the end-to-end evaluation of script knowledge. MCScript2.0 contains approx. 20,000 questions on approx. 3,500 texts, crowdsourced based on a new collection process that results in challenging questions. Half of the questions cannot be answered from the reading texts, but require the use of commonsense and, in particular, script knowledge. We give a thorough analysis of our corpus and show that while the task is not challenging to humans, existing machine comprehension models fail to perform well on the data, even if they make use of a commonsense knowledge base. The dataset is available at http://www.sfb1102. uni-saarland.de/?page_id=2582 1 Introduction People have access to a wide range of common- sense knowledge that is naturally acquired during their lifetime. They make frequent use of such knowledge while speaking to each other, which makes communication highly efficient. The con- versation in Example 1 illustrates this: For Max, it is natural to assume that Rachel paid during her restaurant visit, even if this fact was not mentioned by Rachel. (1) Rachel: “Yesterday, I went to this new Ar- gentinian restaurant to have dinner. I en- joyed a tasty steak.” Max: “What did you pay?” Script knowledge is one of the most important types of commonsense knowledge and subsumes such information (Schank and Abelson, 1977). It is defined as knowledge about everyday situations, also referred to as scenarios. It subsumes informa- tion about the actions that take place during such situations, and their typical temporal order, referred to as events, as well as the persons and objects that typically play a role in the situation, referred to as participants. In the example, script knowledge subsumes the fact that the paying event is a part of the eating in a restaurant scenario. Recent years have seen different approaches to learning script knowledge, centered around two strands: Work around narrative chains that are learned from large text collections (Chambers and Jurafsky, 2008, 2009), and the manual induction of script knowledge via crowdsourcing (Regneri et al., 2010; Wanzare et al., 2016). Script knowl- edge has been represented both symbolically (Jans et al., 2012; Pichotta and Mooney, 2014; Rudinger et al., 2015) and with neural models (Modi and Titov, 2014; Pichotta and Mooney, 2016). Scripts have been evaluated mostly intrinsically (Wanzare et al., 2017; Ostermann et al., 2017). An exception is MCScript (Ostermann et al., 2018a), a reading comprehension corpus with a focus on script knowl- edge, and a predecessor to the data set presented in this work. Previous work has shown, however, that script knowledge is not required for perform- ing well on the data set (Ostermann et al., 2018b). Hence, to date, there exists no evaluation method that allows one to systematically …

Content truncated to 500 words