Latent Retrieval for Weakly Supervised Open Domain Question Answering

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6086 Latent Retrieval for Weakly Supervised Open Domain Question Answering Kenton Lee Ming-Wei Chang Google ResearchKristina Toutanova Seattle, WA {kentonl,mingweichang,kristout}@google.com Abstract Recent work on open domain question answer- ing (QA) assumes strong supervision of the supporting evidence and/or assumes a black- box information retrieval (IR) system to re- trieve evidence candidates. We argue that both are suboptimal, since gold evidence is not al- ways available, and QA is fundamentally dif- ferent from IR. We show for the first time that it is possible to jointly learn the retriever and reader from question-answer string pairs and without any IR system. In this setting, evi- dence retrieval from all of Wikipedia is treated as a latent variable. Since this is impracti- cal to learn from scratch, we pre-train the re- triever with an Inverse Cloze Task. We evalu- ate on open versions of five QA datasets. On datasets where the questioner already knows the answer, a traditional IR system such as BM25 is sufficient. On datasets where a user is genuinely seeking an answer, we show that learned retrieval is crucial, outperforming BM25 by up to 19 points in exact match. 1 Introduction Due to recent advances in reading comprehension systems, there has been a revival of interest in open domain question answering (QA), where the evidence must be retrieved from an open corpus, rather than being given as input. This presents a more realistic scenario for practical applications. Current approaches require a blackbox informa- tion retrieval (IR) system to do much of the heavy lifting, even though it cannot be fine-tuned on the downstream task. In the strongly supervised set- ting popularized by DrQA (Chen et al., 2017), they also assume a reading comprehension model trained on question-answer-evidence triples, such as SQuAD (Rajpurkar et al., 2016). The IR sys- tem is used at test time to generate evidence candi- dates in place of the gold evidence. In the weakly supervised setting, proposed by TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), and Quasar (Dhingra et al., 2017), the dependency on strong supervision is removed by assuming that the IR system provides noisy gold evidence. These approaches rely on the IR system to mas- sively reduce the search space and/or reduce spu- rious ambiguity. However, QA is fundamentally different from IR (Singh, 2012). Whereas IR is concerned with lexical and semantic matching, questions are by definition under-specified and re- quire more language understanding, since users are explicitly looking for unknown information. Instead of being subject to the recall ceiling from blackbox IR systems, we should directly learn to retrieve using question-answering data. In this work, we introduce the first Open- Retrieval Question Answering system (ORQA). ORQA learns to retrieve evidence from an open corpus, and is supervised only by question- answer string pairs. While recent work on im- proving evidence retrieval has made …

Content truncated to 500 words