End-to-End Training of Neural Retrievers for Open-Domain Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6648–6662 August 1–6, 2021. ©2021 Association for Computational Linguistics 6648 End-to-End Training of Neural Retrievers for Open-Domain Question Answering Devendra Singh Sachan1,2∗, Mostofa Patwary3, Mohammad Shoeybi3, Neel Kant3, Wei Ping3, William L Hamilton1,2,4, Bryan Catanzaro3 1Mila - Quebec AI Institute; 2McGill University; 3NVIDIA; 4Canada CIFAR AI Chair sachande@mila.quebec, mpatwary@nvidia.com Abstract Recent work on training neural retrievers for open-domain question answering (OpenQA) has employed both supervised and unsuper- vised approaches. However, it remains un- clear how unsupervised and supervised meth- ods can be used most effectively for neural retrievers. In this work, we systematically study retriever pre-training. We first propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs. This approach leads to absolute gains of 2+ points over the previ- ous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets. We next explore two approaches for end-to- end training of the reader and retriever compo- nents in OpenQA models, which differ in the manner the reader ingests the retrieved docu- ments. Our experiments demonstrate the ef- fectiveness of these approaches as we obtain state-of-the-art results. On the Natural Ques- tions dataset, we obtain a top-20 retrieval ac- curacy of 84%, an improvement of 5 points over the recent DPR model. We also showcase good results on answer extraction, outperform- ing recent models such as REALM and RAG by 3+ points. Our code is available at: https: //github.com/NVIDIA/Megatron-LM. 1 Introduction The task of open-domain question answering (OpenQA) consists of finding answers to the information-seeking questions using a large knowl- edge source such as Wikipedia. This knowledge source is also referred to as evidence and it typically contains millions of documents. Most approaches for OpenQA consist of a two-stage pipeline (Chen et al., 2017; Chen, 2018). In the first stage, given ∗ This work was done during an internship at NVIDIA. Corresponding authors: Devendra Sachan, Mostofa Patwary. Document 1: Bowling is a target sport and recreational activity in which a player rolls a ball towards pins (in pin bowling) or another target (in target bowling) … Document 2: “Hall of Fame” is a song by Irish pop rock band the Script. It is the lead single from their studio album #3. The track features American hip-hop artist will.i.am of The Black Eyed Peas. Document 3: The World Bowling Writers (WBW) International Bowling Hall of Fame was established in 1993 and is located … on the International Bowling campus in Arlington, Texas. Question: Where is the bowling hall of fame located? Stage 1: Retriever Stage 2: Reader Answer: Arlington, Texas Figure 1: An example illustrating OpenQA pipeline. a question, a retriever module identifies the most relevant documents, which is often a very small subset of the evidence known as context documents. Traditionally, approaches based on document rank- ing such as BM25 (Robertson and Zaragoza, …

Content truncated to 500 words