Putting the Horse before the Cart: A Generator-Evaluator Framework for Question Generation from Text

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 812–821 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 812 Putting the Horse Before the Cart: A Generator-Evaluator Framework for Question Generation from Text Vishwajeet Kumar1,2,3, Ganesh Ramakrishnan2, and Yuan-Fang Li3 1IITB-Monash Research Academy, Mumbai, India 2IIT Bombay, Mumbai, India 3Monash University, Melbourne, Australia Abstract Automatic question generation (QG) is a use- ful yet challenging task in NLP. Recent neural network-based approaches represent the state- of-the-art in this task. In this work, we attempt to strengthen them significantly by adopting a holistic and novel generator-evaluator frame- work that directly optimizes objectives that re- ward semantics and structure. The generator is a sequence-to-sequence model that incorpo- rates the structure and semantics of the ques- tion being generated. The generator predicts an answer in the passage that the question can pivot on. Employing the copy and coverage mechanisms, it also acknowledges other con- textually important (and possibly rare) key- words in the passage that the question needs to conform to, while not redundantly repeat- ing words. The evaluator model evaluates and assigns a reward to each predicted ques- tion based on its conformity to the structure of ground-truth questions. We propose two novel QG-specific reward functions for text conformity and answer conformity of the gen- erated question. The evaluator also employs structure-sensitive rewards based on evalua- tion measures such as BLEU, GLEU, and ROUGE-L, which are suitable for QG. In con- trast, most of the previous works only optimize the cross-entropy loss, which can induce in- consistencies between training (objective) and testing (evaluation) measures. Our evaluation shows that our approach significantly outper- forms state-of-the-art systems on the widely- used SQuAD benchmark as per both automatic and human evaluation. 1 Introduction Automatic question generation (QG) is a very im- portant yet challenging problem in NLP. It is de- fined as the task of generating syntactically cor- rect, semantically sound and relevant questions from various input formats such as text, a struc- tured database or a knowledge base (Mannem et al., 2010). More recently, neural network based techniques such as sequence-to-sequence (Seq2Seq) learning have achieved remarkable suc- cess on various NLP tasks, including question generation. A recent deep learning approach to question generation (Serban et al., 2016) in- vestigates a simpler task of generating questions only from a triplet of subject, relation and ob- ject. Learning to ask (referred to as L2A here- inafter) (Du et al., 2017) proposes a Seq2Seq model with attention for question generation from text. (Song et al., 2018) (in an approach re- ferred to as NQGLC hereafter) encoded ground- truth answers and employed bi-directional LSTMs in a Seq2Seq setting. In addition, they use the copy mechanism (See et al., 2017) and context matching to capture interactions between the given ground-truth answer and its context within the pas- sage. In the context of QG from paragraphs, (Zhao et al., 2018) proposed maxout pointer network to keep track of word coverage. Our previous work (Kumar et al., …

Content truncated to 500 words