Training Question Answering Models From Synthetic Data

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5811–5826, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5811 Training Question Answering Models From Synthetic Data Raul Puri‡∗, Ryan Spring§, Mohammad Shoeybi‡, Mostofa Patwary‡∗, Bryan Catanzaro‡ ‡NVIDIA, Santa Clara, California, USA §Rice University, Houston, Texas, USA. Abstract Question and answer generation is a data aug- mentation method that aims to improve ques- tion answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking ad- vantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQUAD1.1 question answering task, we achieve higher ac- curacy using solely synthetic questions and an- swers than when using the SQUAD1.1 train- ing set questions alone. Removing access to real Wikipedia data, we synthesize ques- tions and answers from a synthetic text corpus generated by an 8.3 billion parameter GPT-2 model and achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQUAD1.1 dev set. We further apply our methodology to SQUAD2.0 and show a 2.8 absolute gain on EM score com- pared to prior work using synthetic data. 1 Introduction One of the limitations of developing models for question answering, or any Deep Learning applica- tion for that matter, is the availability and cost of labeled training data. A common approach to alle- viate this need is semi-supervised learning, wherein one trains a model on existing data and uses it to label more data for training (Zhu, 2005; Chapelle et al., 2009; Zhu and Goldberg, 2009; Kingma et al., 2014). This technique has demonstrated benefits in recent literature for image classification (Xie et al., 2019) and question answering (QA) tasks (Alberti et al., 2019a; Dong et al., 2019). However, the complexities of generating questions and answers ∗ Corresponding authors: Raul Puri raulpuric@berkeley.edu, and Mostofa Patwary mpatwary@nvidia.com Text Albert Einstein is known for his theories of special relativity and general relativity. He also made impor- tant contributions to statistical mechanics, especially his mathematical treatment of Brownian motion, his resolution of the paradox of specific heats, and his connection of fluctuations and dissipation. Despite his reservations about its interpretation, Einstein also made contributions to quantum mechanics and, indi- rectly, quantum field theory, primarily through his theoretical studies of the photon. 117M Which two concepts made Einstein’s post on quan- tum mechanics relevant? 768M Albert Einstein also made significant contributions to which field of theory? 8.3B Because of his work with the photon, what theory did he indirectly contribute to? Human What theory did Einstein have reservations about? Table 1: Questions generated by models of increasing capacity with the ground truth answer highlighted in bold. As model size grows, question quality becomes increasingly coherent, complex, and factually relevant. in natural language proves challenging for exist- ing methods, with a large gap in quality remaining between synthetic …

Content truncated to 500 words