End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5445–5460, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5445 End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems Siamak Shakeri†∗, Cicero Nogueira dos Santos∗, Henry Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, Bing Xiang AWS AI New York City, NY {cignog,henghui,patricng,nanfen}@amazon.com {zhiguow,rnallapa,bxiang}@amazon.com siamaks@google.com Abstract We propose an end-to-end approach for syn- thetic QA data generation. Our model com- prises a single transformer-based encoder- decoder network that is trained end-to-end to generate both answers and questions. In a nut- shell, we feed a passage to the encoder and ask the decoder to generate a question and an an- swer token-by-token. The likelihood produced in the generation process is used as a filtering score, which avoids the need for a separate fil- tering model. Our generator is trained by fine- tuning a pretrained LM using maximum likeli- hood estimation. The experimental results in- dicate significant improvements in the domain adaptation of QA models outperforming cur- rent state-of-the-art methods. 1 Introduction Improving question answering (QA) systems through automatically generated synthetic data is a long standing research goal (Mitkov and Ha, 2003; Rus et al., 2010). Although many past works have proposed different strategies for question genera- tion, they have limited or no success in improving the downstream QA task (Du et al., 2017; Sun et al., 2018; Song et al., 2018; Klein and Nabi, 2019; Wang et al., 2020; Ma et al., 2020; Chen et al., 2020; Tuan et al., 2019). Some recent approaches for synthetic QA data generation based on large pretrained language mod- els (LM) have started to demonstrate success in improving the downstream Reading Comprehen- sion (RC) task with automatically generated data (Alberti et al., 2019; Puri et al., 2020). However, these approaches typically consist of multi-stage systems that use three modules: span/answer de- tector, question generator and question filtering. ∗*equal contribution. † Siamak Shakeri is currently with Google. The work was done when he was at AWS AI. Given an input passage, the span detector is respon- sible for extracting spans that will serve as answers for which questions will be generated. This mod- ule normally combines a pretrained QA model with handcrafted heuristics. The question generator is a large LM fine-tuned for the task of conditional generation of questions given passage and answer. The question filtering comprises another RC model that is used to score and filter the generated QA pairs. Each module of this synthetic data genera- tion pipeline is trained/tuned separately and errors from one stage can propagate to the next stages. Additionally, each module is expensive to be com- puted because all use large transformer networks (Vaswani et al., 2017). In this work, we propose an end-to-end approach for synthetic QA data generation. Our model com- prises a single transformer-based encoder-decoder network that is trained end-to-end to generate both the answer and the question. In a nutshell, we feed a passage to the encoder and ask the …

Content truncated to 500 words