Regularization of Distinct Strategies for Unsupervised Question Generation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3266–3277 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3266 Regularization of Distinct Strategies for Unsupervised Question Generation Junmo Kang∗ Giwon Hong∗ Haritz Puerto San Roman∗ Sung-Hyon Myaeng School of Computing, KAIST Daejeon, Republic of Korea {junmo.kang, gch02518, haritzpuerto, myaeng}@kaist.ac.kr Abstract Unsupervised question answering (UQA) has been proposed to avoid the high cost of cre- ating high-quality datasets for QA. One ap- proach to UQA is to train a QA model with questions generated automatically. However, the generated questions are either too sim- ilar to a word sequence in the context or too drifted from the semantics of the context, thereby making it difficult to train a robust QA model. We propose a novel regulariza- tion method based on teacher-student architec- ture to avoid bias toward a particular question generation strategy and modulate the process of generating individual words when a ques- tion is generated. Our experiments demon- strate that we have achieved the goal of gener- ating higher-quality questions for UQA across diverse QA datasets and tasks. We also show that this method can be useful for creating a QA model with few-shot learning. 1 Introduction Machine Reading for Question Answering (MRQA) is the task of answering questions from a context that contains the answer. This field has seen remarkable progress in recent years, with QA models outperforming humans on a question answering (QA) benchmark like SQuAD (Rajpurkar et al., 2016). Training a QA model requires a large amount of data, and constructing such a dataset is usu- ally laborious or sometimes even impossible for some domains and languages. Because of this, Lewis et al. (2019) explored unsupervised ques- tion answering (UQA), a setting where manually constructed triples, (context, question, answer), are not available for training. They approached the problem with unsupervised question generation ∗Equal contribution. Context ... Level 1 of DDM Architecture was formally published in 1986. ... Generated Question LM-type Copy-type When did the first level 1 of DDM Architecture come out? When level 1 of DDM Architecture was formally published? Figure 1: An example of LM-type (left) and copy-type (right) questions generated from a context. Newly gen- erated tokens are in italics and copied tokens in bold. (UQG), where answers are first extracted from con- texts, and then questions are generated from (con- texts, answers) pairs. Such questions are combined into (contexts, questions, answers) triples to form a training dataset for a QA model. Another approach to UQG is based on language models (LM). Radford et al. (2019) proposed an LM approach, GPT-2, that shook the natural lan- guage processing (NLP) research community with its remarkable capability of automatically generat- ing high-quality text. Naturally, GPT-2 was shown to generate high-quality questions from contexts (Klein and Nabi, 2019; Puri et al., 2020). Despite the recent progress made by these efforts, UQG remains to be an open problem. For exam- ple, the method proposed by Lewis et al. (2019) generates a question that is often so …

Content truncated to 500 words