Deep Questions without Deep Understanding

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 889–898, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Deep Questions without Deep Understanding Igor Labutov Sumit Basu Lucy Vanderwende Cornell University Microsoft Research Microsoft Research 124 Hoy Road One Microsoft Way One Microsoft Way Ithaca, NY Redmond, WA Redmond, WA iil4@cornell.edu sumitb@microsoft.com lucyv@microsoft.com Abstract We develop an approach for generating deep (i.e, high-level) comprehension questions from novel text that bypasses the myriad challenges of creating a full se- mantic representation. We do this by de- composing the task into an ontology- crowd-relevance workflow, consisting of first representing the original text in a low-dimensional ontology, then crowd- sourcing candidate question templates aligned with that space, and finally rank- ing potentially relevant templates for a novel region of text. If ontological labels are not available, we infer them from the text. We demonstrate the effectiveness of this method on a corpus of articles from Wikipedia alongside human judgments, and find that we can generate relevant deep questions with a precision of over 85% while maintaining a recall of 70%. 1 Introduction Questions are a fundamental tool for teachers in assessing the understanding of their students. Writing good questions, though, is hard work, and harder still when the questions need to be deep (i.e., high-level) rather than factoid-oriented. These deep questions are the sort of open-ended queries that require deep thinking and recall rather than a rote response, that span significant amounts of content rather than a single sentence. Unsur- prisingly, it is these deep questions that have the greatest educational value (Anderson, 1975; An- dre, 1979; McMillan, 2001). They are thus a key assessment mechanism for a spectrum of online educational options, from MOOCs to interactive tutoring systems. As such, the problem of auto- matic question generation has long been of inter- est to the online education community (Mitkov and Ha, 2003; Schwartz, 2004), both as a means of providing self-assessments directly to students and as a tool to help teachers with question author- ing. Much work to date has focused on questions based on a single sentence of the text (Becker et al., 2012; Lindberg et al., 2013; Mazidi and Niel- sen, 2014), and the ideal of creating deep, concep- tual questions has remained elusive. In this work, we hope to take a significant step towards this challenge by approaching the problem in a some- what unconventional way. Figure 1: Overview of our ontology-crowd-rele- vance approach. While one might expect the natural path to gener- ating deep questions to involve first extracting a semantic representation of the entire text, the state-of-the-art in this area is at too early a stage to achieve such a representation effectively. Ra- ther we take a step back from full understanding, and instead propose an ontology-crowd-relevance workflow for generating high-level questions, shown in Figure 1. This involves 1) decomposing a text into a meaningful, intermediate, low-dimen- sional ontology, 2) soliciting …

Content truncated to 500 words