Exploring Markov Logic Networks for Question Answering

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 685–694, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Exploring Markov Logic Networks for Question Answering Tushar Khot∗, Niranjan Balasubramanian+, Eric Gribkoff$, Ashish Sabharwal∗, Peter Clark∗, Oren Etzioni∗ ∗Allen Institute for AI, +Stony Brook University, $University of Washington {tushark,ashishs,peterc,orene}@allenai.org, niranjan@cs.stonybrook.edu, eagribko@cs.washington.edu Abstract Elementary-level science exams pose sig- nificant knowledge acquisition and rea- soning challenges for automatic question answering. We develop a system that rea- sons with knowledge derived from text- books, represented in a subset of first- order logic. Automatic extraction, while scalable, often results in knowledge that is incomplete and noisy, motivating use of reasoning mechanisms that handle uncer- tainty. Markov Logic Networks (MLNs) seem a natural model for expressing such knowl- edge, but the exact way of leveraging MLNs is by no means obvious. We in- vestigate three ways of applying MLNs to our task. First, we simply use the extracted science rules directly as MLN clauses and exploit the structure present in hard con- straints to improve tractability. Second, we interpret science rules as describing prototypical entities, resulting in a drasti- cally simplified but brittle network. Our third approach, called Praline, uses MLNs to align lexical elements as well as define and control how inference should be per- formed in this task. Praline demonstrates a 15% accuracy boost and a 10x reduction in runtime as compared to other MLN- based methods, and comparable accuracy to word-based baseline approaches. 1 Introduction We consider the problem of answering questions in standardized science exams (Clark et al., 2013), which are used as a benchmark for developing knowledge acquisition and reasoning capabilities. The 4th grade science exam dataset from Clark et al. (2013) tests for a wide variety of knowledge and its application to specific scenarios. In partic- ular we focus on a subset that test students’ un- derstanding of various kinds of general rules and principles (e.g., gravity pulls objects towards the Earth) and their ability to apply these rules to reason about specific situations or scenarios (e.g., which force is responsible for a ball to drop?). Answering these questions can be naturally for- mulated as a reasoning task given the appropri- ate form of knowledge. Prior work on reasoning based approaches has largely relied on manually input knowledge (Lenat, 1995). We present an in- vestigation of a reasoning approach that operates over knowledge automatically extracted from text. In order to effectively reason over knowledge derived from text, a QA system must handle in- complete and potentially noisy knowledge, and allow for reasoning under uncertainty. We cast QA as a reasoning problem in weighted-first order logic. While many probabilistic formalisms ex- ist, we use Markov Logic Networks for the ease of specification of weighted rules. MLNs have been adopted for many NLP tasks (Singla and Domin- gos, 2006a; Kok and Domingos, 2008; Poon and Domingos, 2009). Recently, Beltagy et al. (2013) and Beltagy and Mooney (2014) have shown that MLNs can be used to reason …

Content truncated to 500 words