Learning What is Essential in Questions

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 80–89, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics CoNLL’17 Learning What is Essential in Questions Daniel Khashabi† Univ. of Pennsylvania danielkh@cis.upenn.edu Tushar Khot Ashish Sabharwal Allen Institute for AI tushark,ashishs@allenai.org Dan Roth† Univ. of Pennsylvania danroth@cis.upenn.edu Abstract Question answering (QA) systems are eas- ily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains. This paper introduces and studies the notion of essential ques- tion terms with the goal of improving such QA solvers. We illustrate the im- portance of essential question terms by showing that humans’ ability to answer questions drops significantly when essen- tial terms are eliminated from questions. We then develop a classifier that reliably (90% mean average precision) identifies and ranks essential terms in questions. Fi- nally, we use the classifier to demonstrate that the notion of question term essen- tiality allows state-of-the-art QA solvers for elementary-level science questions to make better and more informed decisions, improving performance by up to 5%. We also introduce a new dataset of over 2,200 crowd-sourced essential terms anno- tated science questions. 1 Introduction Understanding what a question is really about is a fundamental challenge for question answering systems that operate with a natural language in- terface. In domains with multi-sentence ques- tions covering a wide array of subject areas, such as standardized tests for elementary level science, the challenge is even more pronounced (Clark, 2015). Many QA systems in such domains † Most of the work was done when the first and last authors were affiliated with the University of Illinois, Urbana- Champaign. derive significant leverage from relatively shal- low Information Retrieval (IR) and statistical cor- relation techniques operating on large unstruc- tured corpora (Kwok et al., 2001; Clark et al., 2016). Inference based QA systems operating on (semi-)structured knowledge formalisms have also demonstrated complementary strengths, by using optimization formalisms such as Semantic Pars- ing (Yih et al., 2014), Integer Linear Program (ILP) (Khashabi et al., 2016), and probabilistic logic formalisms such as Markov Logic Networks (MLNs) (Khot et al., 2015). These QA systems, however, often struggle with seemingly simple questions because they are unable to reliably identify which question words are redundant, irrelevant, or even intentionally dis- tracting. This reduces the systems’ precision and results in questionable “reasoning” even when the correct answer is selected among the given alter- natives. The variability of subject domain and question style makes identifying essential question words challenging. Further, essentiality is context dependent—a word like ‘animals’ can be critical for one question and distracting for another. Con- sider the following example: One way animals usually respond to a sudden drop in tem- perature is by (A) sweating (B) shivering (C) blinking (D) salivating. A state-of-the-art optimization based QA system called TableILP (Khashabi et al., 2016), which performs reasoning by aligning the question to semi-structured knowledge, aligns only the word ‘animals’ when answering this question. …

Content truncated to 500 words