Synthetic QA Corpora Generation with Roundtrip Consistency

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168–6173 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6168 Synthetic QA Corpora Generation with Roundtrip Consistency Chris Alberti Daniel Andor Emily Pitler Jacob Devlin Michael Collins Google Research {chrisalberti, andor, epitler, jacobdevlin, mjcollins}@google.com Abstract We introduce a novel method of generating synthetic question answering corpora by com- bining models of question generation and an- swer extraction, and by filtering the results to ensure roundtrip consistency. By pretraining on the resulting corpora we obtain significant improvements on SQuAD2 (Rajpurkar et al., 2018) and NQ (Kwiatkowski et al., 2019), es- tablishing a new state-of-the-art on the latter. Our synthetic data generation models, for both question generation and answer extraction, can be fully reproduced by finetuning a publicly available BERT model (Devlin et al., 2018) on the extractive subsets of SQuAD2 and NQ. We also describe a more powerful variant that does full sequence-to-sequence pretraining for question generation, obtaining exact match and F1 at less than 0.1% and 0.4% from hu- man performance on SQuAD2. 1 Introduction Significant advances in Question Answering (QA) have recently been achieved by pretraining deep transformer language models on large amounts of unlabeled text data, and finetuning the pretrained models on hand labeled QA datasets, e.g. with BERT (Devlin et al., 2018). Language modeling is however just one exam- ple of how an auxiliary prediction task can be con- structed from widely available natural text, namely by masking some words from each passage and training the model to predict them. It seems plau- sible that other auxiliary tasks might exist that are better suited for QA, but can still be constructed from widely available natural text. It also seems intuitive that such auxiliary tasks will be more helpful the closer they are to the particular QA task we are attempting to solve. Based on this intuition we construct auxil- iary tasks for QA, generating millions of syn- Input (C) ... in 1903, boston participated in the first modern world series, going up against the pittsburgh pirates ... (1) C → A 1903 (2) C, A → Q when did the red sox first go to the world series (3) C, Q → A′ 1903 (4) A ?= A′ Yes Table 1: Example of how synthetic question-answer pairs are generated. The model’s predicted answer (A′) matches the original answer the question was generated from, so the example is kept. thetic question-answer-context triples from unla- beled passages of text, pretraining a model on these examples, and finally finetuning on a partic- ular labeled dataset. Our auxiliary tasks are illus- trated in Table 1. For a given passage C, we sample an extrac- tive short answer A (Step (1) in Table 1). In Step (2), we generate a question Q conditioned on A and C, then (Step (3)) predict the extractive an- swer A′ conditioned on Q and C. If A and A′ match we finally emit (C, Q, A) as a …

Content truncated to 500 words