SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 747–757 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge Simon Ostermann Michael Roth Ashutosh Modi Stefan Thater Manfred Pinkal Saarland University Saarbr¨ucken, Germany {simono|mroth|ashutosh|stth|pinkal}@coli.uni-saarland.de Abstract This report summarizes the results of the Se- mEval 2018 task on machine comprehension using commonsense knowledge. For this ma- chine comprehension task, we created a new corpus, MCScript. It contains a high num- ber of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neu- ral approaches. The best performing system achieves an accuracy of 83.95%, outperform- ing the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%. 1 Introduction Developing algorithms for understanding natural language is not trivial. Natural language comes with its own complexity and inherent ambiguities. Ambiguities can occur, for example, at the level of word meaning, syntactic structure, or semantic interpretation. Traditionally, Natural Language Un- derstanding (NLU) systems have resolved ambigui- ties using information from the textual context (e.g. neighboring words and sentences), for example via distributional methods (Lenci, 2008). However, many times context may be absent or may lack sufficient information to resolve the ambiguity. In such cases, it would be beneficial to include com- monsense knowledge about the world in an NLU system. For example, consider example (1). (1) The waitress brought Rachel’s order. She ate the food with great pleasure. Looking at the example in isolation, the person eating the food could be either Rachel or the wait- ress. Using commonsense knowledge, or, more specifically, script knowledge about the RESTAU- RANT scenario, helps to resolve the referent of the pronoun: Rachel ordered the food. The person who orders the food is the customer. So Rachel should eat the food, she thus refers to Rachel. This shared task assesses how the inclusion of commonsense knowledge benefits natural language understanding systems. In particular, we focus on commonsense knowledge about everyday activi- ties, referred to as scripts. Scripts are sequences of events describing stereotypical human activities (also called scenarios), for example baking a cake, taking a bus, etc. (Schank and Abelson, 1975). The concept of scripts has its underpinnings in cognitive psychology and has been shown to be an important component of the human cognitive system (Bower et al., 1979; Schank, 1982; Modi et al., 2017). From an application perspective, scripts have been shown to be useful for a variety of tasks, including story understanding (Schank, 1990), information extrac- tion (Rau et al., 1989), and drawing inferences from texts (Miikkulainen, 1993). Factual knowledge is mentioned explicitly in texts from sources such as Wikipedia and news pa- pers. On the contrary, script knowledge is often im- plicit in the texts as it is assumed to be known to the comprehender. Because of this implicitness, learn- ing script knowledge from texts is very …

Content truncated to 500 words