Exploring Question-Specific Rewards for Generating Deep Questions

Proceedings of the 28th International Conference on Computational Linguistics, pages 2534–2546 Barcelona, Spain (Online), December 8-13, 2020 2534 Exploring Question-Specific Rewards for Generating Deep Questions Yuxi Xie1 Liangming Pan2,4∗ Dongzhe Wang3 Min-Yen Kan4 Yansong Feng1,5 1Wangxuan Institute of Computer Technology, Peking University 2NUS Graduate School for Integrative Sciences and Engineering 3ZhuiYi Technology, Singapore 4School of Computing, National University of Singapore, Singapore 5 The MOE Key Laboratory of Computational Linguistics, Peking University xieyuxi@pku.edu.cn, e0272310@u.nus.edu, wdzethan2010@gmail.com kanmy@comp.nus.edu.sg, fengyansong@pku.edu.cn Abstract Recent question generation (QG) approaches often utilize the sequence-to-sequence framework (Seq2Seq) to optimize the log-likelihood of ground-truth questions using teacher forcing. How- ever, this training objective is inconsistent with actual question quality, which is often reflected by certain global properties such as whether the question can be answered by the document. As such, we directly optimize for QG-specific objectives via reinforcement learning to improve question quality. We design three different rewards that target to improve the fluency, relevance, and answerability of generated questions. We conduct both automatic and human evaluations in addition to a thorough analysis to explore the effect of each QG-specific reward. We find that op- timizing question-specific rewards generally leads to better performance in automatic evaluation metrics. However, only the rewards that correlate well with human judgement (e.g., relevance) lead to real improvement in question quality. Optimizing for the others, especially answerability, introduces incorrect bias to the model, resulting in poor question quality. Our code is publicly available at https://github.com/YuxiXie/RL-for-Question-Generation. 1 Introduction Question Generation (QG) aims to endow machines with the ability to ask relevant and to-the-point questions about a document. QG has important practical applications, such as generating assessments for course materials in education (Heilman and Smith, 2010; Lindberg et al., 2013), prompting user in- teraction in dialog systems (Shukla et al., 2019), enabling machines to ask clarification questions such as FAQs (Saeidi et al., 2018; Krishna and Iyyer, 2019), and automatically building large-scale QA datasets for the research community (Du et al., 2017; Zhao et al., 2018). Recent QG approaches (Du et al., 2017; Zhao et al., 2018; Liu et al., 2019) have used Seq2Seq models with attention (Bahdanau et al., 2015), which feeds the input document into an encoder, and generates a question about the document through a decoder. The training objective is to maximize the log likelihood of the ground-truth question paired with each input document using teacher forcing (Williams and Zipser, 1989). However, as the ground-truth questions are insufficient to account for the many equivalent ways of asking a question, this likelihood-based training suffers from the problem of exposure bias (Ranzato et al., 2016), i.e., the model does not learn how to distribute probability mass over sequences that are valid but different from the ground truth. To address this issue, previous QG works proposed to optimize the model directly on question-specific rewards via Reinforcement Learning (RL). This process decouples the training procedure from the ground truth data, so that the space of possible questions can be better explored. Moreover, it allows the training to target …

Content truncated to 500 words