A Strong Lexical Matching Method for the Machine Comprehension Test

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1693–1698, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Strong Lexical Matching Method for the Machine Comprehension Test Ellery Smith, Nicola Greco, Matko Boˇsnjak, Andreas Vlachos Department of Computer Science University College London {e.smith,n.greco,m.bosnjak,a.vlachos}@cs.ucl.ac.uk Abstract Machine comprehension of text is the overarching goal of a great deal of re- search in natural language processing. The Machine Comprehension Test (Richard- son et al., 2013) was recently proposed to assess methods on an open-domain, exten- sible, and easy-to-evaluate task consisting of two datasets. In this paper we develop a lexical matching method that takes into account multiple context windows, ques- tion types and coreference resolution. We show that the proposed method outper- forms the baseline of Richardson et al. (2013), and despite its relative simplicity, is comparable to recent work using ma- chine learning. We hope that our approach will inform future work on this task. Fur- thermore, we argue that MC500 is harder than MC160 due to the way question an- swer pairs were created. 1 Introduction Machine comprehension of text is the central goal in NLP. The academic community has proposed a variety of tasks, such as information extrac- tion (Sarawagi, 2008), semantic parsing (Mooney, 2007) and textual entailment (Androutsopoulos and Malakasiotis, 2010). However, these tasks as- sess performance on each task individually, rather than on overall progress towards machine compre- hension of text. To this end, Richardson et al. (2013) proposed the Machine Comprehension Test (MCTest), a new challenge that aims at evaluating machine comprehension. It does so through an open- domain multiple-choice question answering task on fictional stories requiring the common sense reasoning typical of a 7-year-old child. It is easy to evaluate as it consists of multiple choice ques- tions. Richardson et al. (2013) also showed how the creation of stories and questions can be crowd- sourced efficiently, constructing two datasets for the task, namely MC160 and MC500. In ad- dition, the authors presented a lexical matching baseline which is combined with the textual en- tailment recognition system BIUTEE (Stern and Dagan, 2011). In this paper we develop an approach based on lexical matching which we extend by taking into account the type of the question and coreference resolution. These components improve the per- formance on questions that are difficult to han- dle with pure lexical matching. When combined with BIUTEE, we achieved 74.27% accuracy on MC160 and 65.96% on MC500, which are signif- icantly better than those reported by Richardson et al. (2013). Despite the simplicity of our ap- proach, these results are comparable with the re- cent machine learning-based approaches proposed by Narasimhan and Barzilay (2015), Wang et al. (2015) and Sachan et al. (2015). Furthermore, we examine the types of questions and answers in the two datasets. We argue that some types are relatively simple to answer, partly due to the limited vocabulary used, which explains why simple lexical matching methods can per- form well. On the other …

Content truncated to 500 words