Learning to Generate Questions by Learning to Recover Answer-containing Sentences

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1516–1529 August 1–6, 2021. ©2021 Association for Computational Linguistics 1516 Learning to Generate Questions by Learning to Recover Answer-containing Sentences Seohyun Back12∗, Akhil Kedia1∗, Sai Chetan Chinthakindi1 , Haejun Lee1 and Jaegul Choo2 1Samsung Research, Seoul, South Korea 2KAIST, Daejeon, South Korea {scv.back,sai.chetan,akhil.kedia,haejun82.lee}@samsung.com jchoo@kaist.ac.kr Abstract To train a question answering model based on machine reading comprehension (MRC), sig- nificant effort is required to prepare annotated training data composed of questions and their answers from contexts. Recent research has focused on synthetically generating a question from a given context and an annotated (or gen- erated) answer by training an additional gener- ative model to augment the training data. In light of this research direction, we propose a novel pre-training approach that learns to gen- erate contextually rich questions, by recover- ing answer-containing sentences. We evalu- ate our method against existing ones in terms of the quality of generated questions, and fine-tuned MRC model accuracy after train- ing on the data synthetically generated by our method. We consistently improve the ques- tion generation capability of existing models such as T5 and UniLM, and achieve state-of- the-art results on MS MARCO and NewsQA, and comparable results to the state-of-the-art on SQuAD. Additionally, the data syntheti- cally generated by our approach is beneficial for boosting up the downstream MRC accu- racy across a wide range of datasets, such as SQuAD-v1.1, v2.0, KorQuAD and BioASQ, without any modification to the existing MRC models. Furthermore, our method shines espe- cially when a limited amount of pre-training or downstream MRC data is given. 1 Introduction Machine reading comprehension (MRC), which finds the answer to a given question from its accom- panying paragraphs (called context), is an essential task in natural language processing. With the re- lease of high-quality human-annotated datasets for the task, such as SQuAD-v1.1 (Rajpurkar et al., 2016), -v2.0 (Rajpurkar et al., 2018), and Ko- ∗These authors contributed equally. rQuAD (Lim et al., 2019), researchers have pro- posed MRC models even surpassing human scores. These datasets commonly involve finding a snippet within a context as an answer to a given question. However, these datasets require significant amount of human effort to create questions and their relevant answers from given contexts. Of- ten the size of the annotated data is relatively small compared to that of data used in other self- supervised tasks such as language modeling, limit- ing the accuracy. To overcome this issue, researchers have studied models for generating synthetic questions from a given context along with annotated (or generated) answers on large corpora such as Wikipedia. Golub et al. (2017) suggested a two-stage network of gen- erating question-answer pairs which first chooses answers conditioned on the paragraph and then generates a question conditioned on the chosen an- swer. Dong et al. (2019) showed that pre-training on unified language modeling from large corpora including Wikipedia improves the question gener- ation capability. Alberti et al. (2019) introduced a self-supervised pre-training technique for question generation via …

Content truncated to 500 words