Mind the Gap: Learning to Choose Gaps for Question Generation

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 742–751, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Mind the Gap: Learning to Choose Gaps for Question Generation Lee Becker Department of Computer Science University of Colorado Boulder Boulder, CO 80309 lee.becker@colorado.edu Sumit Basu and Lucy Vanderwende Microsoft Research One Microsoft Way Redmond, WA 98052 {sumitb,lucyv}@microsoft.com Abstract Not all learning takes place in an educational setting: more and more self-motivated learners are turning to on-line text to learn about new topics. Our goal is to provide such learners with the well-known benefits of testing by au- tomatically generating quiz questions for on- line text. Prior work on question generation has focused on the grammaticality of generat- ed questions and generating effective multi- ple-choice distractors for individual question targets, both key parts of this problem. Our work focuses on the complementary aspect of determining what part of a sentence we should be asking about in the first place; we call this “gap selection.” We address this problem by asking human judges about the quality of questions generated from a Wikipedia-based corpus, and then training a model to effective- ly replicate these judgments. Our data shows that good gaps are of variable length and span all semantic roles, i.e., nouns as well as verbs, and that a majority of good questions do not focus on named entities. Our resulting system can generate fill-in-the-blank (cloze) ques- tions from generic source materials. 1 Introduction Assessment is a fundamental part of teaching, both to measure a student’s mastery of the material and to identify areas where she may need reinforce- ment or additional instruction. Assessment has also been shown an important part of learning, as test- ing assists retention and can be used to guide learn- ing (Anderson & Biddle, 1975). Thus, as learners move on from an educational setting to unstruc- tured self-learning settings, they would still benefit from having the means for assessment available. Even in traditional educational settings, there is a need for automated test generation, as teachers want multiple tests for topics to give to different students, and students want different tests with which to study and practice the material. One possible solution to providing quizzes for new source material is the automatic generation of questions. This is a task the NLP community has already embraced, and significant progress has been made in recent years with the introduction of a shared task (Rus et al., 2010). However, thus far the research community has focused on the prob- lem of generating grammatical questions (as in Heilman and Smith (2010a)) or generating effec- tive distractors for multiple-choice questions (Agarwal and Mannem, 2011). While both of these research threads are of crit- ical importance, there is another key issue that must be addressed – which questions should we be asking in the first place? We have highlighted this aspect of the problem in the past (see Vanderwende (2008)) and begin to address it in …

Content truncated to 500 words