Good Question! Statistical Ranking for Question Generation

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 609–617, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Good Question! Statistical Ranking for Question Generation Michael Heilman Noah A. Smith Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA {mheilman,nasmith}@cs.cmu.edu Abstract We address the challenge of automatically generating questions from reading materials for educational practice and assessment. Our approach is to overgenerate questions, then rank them. We use manually written rules to perform a sequence of general purpose syn- tactic transformations (e.g., subject-auxiliary inversion) to turn declarative sentences into questions. These questions are then ranked by a logistic regression model trained on a small, tailored dataset consisting of labeled output from our system. Experimental results show that ranking nearly doubles the percentage of questions rated as acceptable by annotators, from 27% of all questions to 52% of the top ranked 20% of questions. 1 Introduction In this paper, we focus on question generation (QG) for the creation of educational materials for read- ing practice and assessment. Our goal is to gener- ate fact-based questions about the content of a given article. The top-ranked questions could be filtered and revised by educators, or given directly to stu- dents for practice. Here we restrict our investigation to questions about factual information in texts. We begin with a motivating example. Consider the following sentence from the Wikipedia article on the history of Los Angeles:1 During the Gold Rush years in northern California, Los Angeles became known as the “Queen of the Cow Counties” for its role in supplying beef and other foodstuffs to hungry miners in the north. Consider generating the following question from that sentence: What did Los Angeles become known 1“History of Los Angeles.” Wikipedia. 2009. Wikimedia Foundation, Inc. Retrieved Nov. 17, 2009 from: http://en. wikipedia.org/wiki/History_of_Los_Angeles. as the “Queen of the Cow Counties” for? We observe that the QG process can be viewed as a two-step process that essentially “factors” the problem into simpler components.2 Rather than si- multaneously trying to remove extraneous informa- tion and transform a declarative sentence into an in- terrogative one, we first transform the input sentence into a simpler sentence such as Los Angeles become known as the “Queen of the Cow Counties” for its role in supplying beef and other foodstuffs to hungry miners in the north, which we then can then trans- form into a more succinct question. Question transformation involves complex long distance dependencies. For example, in the ques- tion about Los Angeles, the word what at the begin- ning of the sentence is a semantic argument of the verb phrase known as ... at the end of the ques- tion. The characteristics of such phenomena are (ar- guably) difficult to learn from corpora, but they have been studied extensively in linguistics (Ross, 1967; Chomsky, 1973). We take a rule-based approach in order to leverage this linguistic knowledge. However, since many phenomena pertaining to question generation are not so easily encoded with rules, …

Content truncated to 500 words