Unsupervised Question Answering by Cloze Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4896–4910 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4896 Unsupervised Question Answering by Cloze Translation Patrick Lewis Facebook AI Research University College London plewis@fb.com Ludovic Denoyer Facebook AI Research denoyer@fb.com Sebastian Riedel Facebook AI Research University College London sriedel@fb.com Abstract Obtaining training data for Question Answer- ing (QA) is time-consuming and resource- intensive, and existing QA datasets are only available for limited domains and languages. In this work, we explore to what extent high quality training data is actually required for Extractive QA, and investigate the possibility of unsupervised Extractive QA. We approach this problem by first learning to generate con- text, question and answer triples in an unsu- pervised manner, which we then use to syn- thesize Extractive QA training data automati- cally. To generate such triples, we first sam- ple random context paragraphs from a large corpus of documents and then random noun phrases or named entity mentions from these paragraphs as answers. Next we convert an- swers in context to “fill-in-the-blank” cloze questions and finally translate them into nat- ural questions. We propose and compare var- ious unsupervised ways to perform cloze-to- natural question translation, including train- ing an unsupervised NMT model using non- aligned corpora of natural questions and cloze questions as well as a rule-based approach. We find that modern QA models can learn to an- swer human questions surprisingly well using only synthetic training data. We demonstrate that, without using the SQuAD training data at all, our approach achieves 56.4 F1 on SQuAD v1 (64.5 F1 when the answer is a Named en- tity mention), outperforming early supervised models. 1 Introduction Extractive Question Answering (EQA) is the task of answering questions given a context document under the assumption that answers are spans of to- kens within the given document. There has been substantial progress in this task in English. For SQuAD (Rajpurkar et al., 2016), a common EQA benchmark dataset, current models beat human The London Sevens is a rugby tournament held at Twickenham Stadium in London. It is part of the World Rugby Sevens Series. For many years the London Sevens was the last tournament of each season but the Paris Sevens became the last stop on the calendar in 2018. Question Answering Cloze Translation Cloze Generation QA Model the Paris sevens become the last stop on the calendar in MASK Question Generation 2018 Answer Extraction Context Cloze Question Natural Question Answer When did the Paris Sevens become the last stop on the calendar? Figure 1: A schematic of our approach. The right side (dotted arrows) represents traditional EQA. We intro- duce unsupervised data generation (left side, solid ar- rows), which we use to train standard EQA models performance; For SQuAD 2.0 (Rajpurkar et al., 2018), ensembles based on BERT (Devlin et al., 2018) now match human performance. Even for the recently introduced Natural Questions cor- pus (Kwiatkowski et al., 2019), human perfor- mance …

Content truncated to 500 words