A Feasibility Study of Answer-Agnostic Question Generation for Education

Findings of the Association for Computational Linguistics: ACL 2022, pages 1919 - 1926 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Feasibility Study of Answer-Agnostic Question Generation for Education Liam Dugan, Eleni Miltsakaki, Shriyash Upadhyay, Etan Ginsberg, Hannah Gonzalez, Dayheon Choi, Chuning Yuan, Chris Callison-Burch University of Pennsylvania {ldugan,elenimi,shriyash,etangins,hannahgl,dhachoi,dianacny,ccb}@seas.upenn.edu Abstract We conduct a feasibility study into the applica- bility of answer-agnostic question generation models to textbook passages. We show that a significant portion of errors in such systems arise from asking irrelevant or uninterpretable questions and that such errors can be amelio- rated by providing summarized input. We find that giving these models human-written sum- maries instead of the original text results in a significant increase in acceptability of gener- ated questions (33% → 83%) as determined by expert annotators. We also find that, in the absence of human-written summaries, au- tomatic summarization can serve as a good middle ground. 1 Introduction Writing good questions that target salient concepts is difficult and time consuming. Automatic Ques- tion Generation (QG) is a powerful tool that could be used to significantly lessen the amount of time it takes to write such questions. A QG system that au- tomatically generates relevant questions from text- books would help professors write quizzes faster and help students stay engaged when reviewing course material. Previous work on QG has focused primarily on answer-aware QG models. These models require the explicit selection of an answer span in the in- put context, typically through the usage of high- light tokens. This adds significant overhead to the question generation process and is undesirable in cases where clear lists of salient key terms are un- available. We conduct a feasibility study1 on the application of answer-agnostic question generation models (ones which do not require manual selec- tion of answer spans) to an educational context. Our contributions are as follows: 1The data collected and software used is available at https://github.com/liamdugan/summary-qg Input: The perplexity of a language model on a test set is the inverse probability of the test set, normalized by the number of words. For a test set W = w1w2…wN we can use the chain rule to expand the probability of W. Automatic Summarization Human Summarization Q: What is the perplexity of a test set? A: w1w2…wN Q: What is the perplexity of a language model on a test set? A: the inverse probability of the test set Q: What is the inverse probability of the test set normalized by the number of words? A: Perplexity Figure 1: Relevance, interpretability, and acceptabil- ity of generated questions are significantly improved when using human-written summaries (yellow) or automatically-generated summaries (green) as input in- stead of the original text (red). • We show that the primary way answer- agnostic QG models fail is by generating irrel- evant or uninterpretable questions. • We show that giving answer-agnostic QG models human-written summaries instead of the original text results in significant increases in question acceptability (33% → 83%), rel- evance (61% → 95%), and in-context …

Content truncated to 500 words