Evaluating Rewards for Question Generation Models

Evaluating Rewards for Question Generation Models Tom Hosking University College London thomas.hosking.17@ucl.ac.uk Sebastian Riedel University College London sriedel@ucl.ac.uk Abstract Recent approaches to question generation have used modifications to a Seq2Seq architecture inspired by advances in machine translation. Models are trained using teacher forcing to optimise only the one-step-ahead prediction. However, at test time, the model is asked to generate a whole sequence, causing errors to propagate through the generation process (ex- posure bias). A number of authors have sug- gested that optimising for rewards less tightly coupled to the training data might counter this mismatch. We therefore optimise directly for various objectives beyond simply replicating the ground truth questions, including a novel approach using an adversarial discriminator that seeks to generate questions that are in- distinguishable from real examples. We con- firm that training with policy gradient meth- ods leads to increases in the metrics used as rewards. We perform a human evaluation, and show that although these metrics have previ- ously been assumed to be good proxies for question quality, they are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source. 1 Introduction Posing questions about a document in natural lan- guage is a crucial aspect of the effort to automati- cally process natural language data, enabling ma- chines to ask clarification questions (Saeidi et al., 2018), become more robust to queries (Yu et al., 2018), and to act as automatic tutors (Heilman and Smith, 2010). Recent approaches to question generation have used Seq2Seq (Sutskever et al., 2014) models with attention (Bahdanau et al., 2014) and a form of copy mechanism (Vinyals et al., 2015; Gulcehre et al., 2016). Such models are trained to generate a plausible question, conditioned on an input docu- ment and answer span within that document (Zhou et al., 2018; Du et al., 2017; Du and Cardie, 2018; Yuan et al., 2017). There are currently no dedicated question generation datasets, and authors have used the context-question-answer triples available in SQuAD (Rajpurkar et al., 2016). Only a sin- gle question is available for each context-answer pair, and models are trained using teacher forc- ing (Williams and Zipser, 1989). This lack of di- verse training data combined with the one-step- ahead training procedure exacerbates the problem of exposure bias (Ranzato et al., 2015). The model does not learn how to distribute probability mass over sequences that are valid but different to the ground truth; during inference, the model must predict the whole sequence, and may not be robust to mistakes during decoding. Recent work has investigated training the mod- els directly on a performance based objective, ei- ther by optimising for BLEU score (Kumar et al., 2018a) or other quality metrics (Yuan et al., 2017). By decoupling the training procedure from the ground truth data, the model is able to explore the space of possible questions and learn to re- cover from suboptimal predictions during decod- ing. While the metrics used seem to be intuitively good choices, there …

Content truncated to 500 words