MITRE at SemEval-2018 Task 11: Commonsense Reasoning without Commonsense Knowledge

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1078–1082 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics MITRE at SemEval-2018 Task 11: Commonsense Reasoning without Commonsense Knowledge Elizabeth M. Merkhofer, John Henderson, David Bloom, Laura Strickhart and Guido Zarrella The MITRE Corporation 202 Burlington Road Bedford, MA 01730-1420, USA {emerkhofer,jhndrsn,dtbloom,lstrickhart,jzarrella}@mitre.org Abstract This paper describes MITRE’s participation in SemEval-2018 Task 11: Machine Comprehen- sion using Commonsense Knowledge. The techniques explored range from simple bag- of-ngrams classifiers to neural architectures with varied attention and alignment mecha- nisms. Logistic regression ties the systems to- gether into an ensemble submitted for evalu- ation. The resulting system answers reading comprehension questions with 82.27% accu- racy. 1 Introduction Reading comprehension tasks measure the ability to answer questions that require inference from a free text story. This SemEval task, like many stan- dardized tests (e.g., the SAT), provides multiple- choice answers. Reading comprehension may rely on information explicitly contained in the text, such as which actors are present, as well as el- ements of world knowledge, like understanding common scripts. Early attempts at statistical reading comprehen- sion include fill-in-the-blank questions and com- bine rich question categorization, information re- trieval techniques, and entity recognition with type-specific, hand-crafted tactics (Hirschman et al., 1999; Anand et al., 2000). More recent neural work uses continuous, dis- tributed semantic space rather than n-gram overlap to find answers similar to the story. Trischler et al. (2016) compute a version of word overlap by find- ing cosine similarity of word representations be- tween sections. Yin et al. (2016) restate the ques- tion and answer and incorporate a question-type classifier. In this effort we explored neural distributed rep- resentations and lexicon-based machine learning approaches, especially attention and word overlap. 2 Task, Data and Evaluation Machine Comprehension using Commonsense Knowledge was a shared task organized within SemEval-2018 (Ostermann et al., 2018b). The task organizers released a dataset of 1,689 stories and 10,872 questions (up to 14 questions per story), split into training and development sets. The stories were first-person, English language narratives written by Mechanical Turk workers in response to prompts asking them to describe a sce- nario, like going on a date. Stories were up to 860 words long, with 90% under 273 words and a me- dian length of 183 words. Questions are up to 22 words long, with a median length of seven. Each question has two possible answers. Answers vary in length from a single word, including yes or no, to 30 words, with a median length of three. Many questions are repeated between multiple stories. A sample story and questions are shown in Figure 1. Dataset construction is detailed in Ostermann et al. (2018a). The evaluation metric for the task is sim- ple accuracy: the portion of correct answers. Question types Table 1 quantifies the question types in the development set. We created this tax- onomy to better understand the dataset. It also al- lowed us to direct our development toward model weaknesses. Common Sense …

Content truncated to 500 words