Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2495–2509, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2495 Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering Shiyue Zhang Mohit Bansal UNC Chapel Hill {shiyue, mbansal}@cs.unc.edu Abstract Text-based Question Generation (QG) aims at generating natural and relevant questions that can be answered by a given answer in some context. Existing QG models suffer from a “semantic drift” problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. In this pa- per, we first propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate seman- tically valid questions. Second, since the tra- ditional evaluation metrics (e.g., BLEU) often fall short in evaluating the quality of gener- ated questions, we propose a QA-based evalu- ation method which measures the QG model’s ability to mimic human annotators in gener- ating QA training data. Experiments show that our method achieves the new state-of-the- art performance w.r.t. traditional metrics, and also performs best on our QA-based evalua- tion metrics. Further, we investigate how to use our QG model to augment QA datasets and enable semi-supervised QA. We propose two ways to generate synthetic QA pairs: gen- erate new questions from existing articles or collect QA pairs from new articles. We also propose two empirically effective strategies, a data filter and mixing mini-batch training, to properly use the QG-generated data for QA. Experiments show that our method improves over both BiDAF and BERT QA baselines, even without introducing new articles.1 1 Introduction In contrast to the rapid progress shown in Ques- tion Answering (QA) tasks (Rajpurkar et al., 2016; Joshi et al., 2017; Yang et al., 2018), the task of Question Generation (QG) remains understudied and challenging. However, as an important dual 1Code and models publicly available at: https:// github.com/ZhangShiyue/QGforQA Context: ...during the age of enlightenment, philoso- phers such as john locke advocated the principle in their writings, whereas others, such as thomas hobbes, strongly opposed it. montesquieu was one of the fore- most supporters of separating the legislature, the exec- utive, and the judiciary... Gt: who was an advocate of separation of powers? Base: who opposed the principle of enlightenment? Ours: who advocated the principle in the age of en- lightenment? Table 1: An examples of the “semantic drift” issue in Question Generation (“Gt” is short for “ground truth”). task to QA, QG can not only be used to aug- ment QA datasets (Duan et al., 2017), but can also be applied in conversation and education systems (Heilman and Smith, 2010; Lindberg et al., 2013). Furthermore, given that existing QA models of- ten fall short by doing simple word/phrase match- ing rather than true comprehension (Jia and Liang, 2017), the task of QG, which usually needs com- plicated semantic reasoning and syntactic varia- tion, should be another way to encourage …

Content truncated to 500 words