Relevance-guided Supervision for OpenQA with ColBERT

Relevance-guided Supervision for OpenQA with ColBERT Omar Khattab Stanford University, United States okhattab@stanford.edu Christopher Potts Stanford University, United States cgpotts@stanford.edu Matei Zaharia Stanford University, United States matei@cs.stanford.edu Abstract Systems for Open-Domain Question An- swering (OpenQA) generally depend on a retriever for finding candidate passages in a large corpus and a reader for extracting answers from those passages. In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages. We argue that this modeling choice is insufficiently expressive for dealing with the complexity of natural language questions. To address this, we define ColBERT-QA, which adapts the scalable neural retrieval model ColBERT to OpenQA. ColBERT creates fine-grained interactions between questions and passages. We propose an efficient weak supervision strategy that iteratively uses ColBERT to create its own training data. This greatly improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA, and the resulting system attains state-of-the-art extractive OpenQA performance on all three datasets. 1 Introduction The goal of Open-Domain Question Answering (OpenQA; Voorhees and Tice, 2000) is to find an- swers to factoid questions in potentially massive unstructured text corpora. Systems for OpenQA typically depend on two major components: a re- trieval model to find passages that are relevant to the user’s question and a machine reading model to try to find an answer to the question in the re- trieved passages. At its best, this should combine the power and scalability of current information retrieval (IR) models with recent advances in ma- chine reading comprehension (MRC). However, if the notions of relevance embedded in the IR model fail to align with the requirements of question an- swering, the MRC model will not reliably see the best passages and the system will perform poorly. Many prior approaches to OpenQA rely on classical IR models (e.g., BM25; Robertson et al., 1995) whose notions of relevance are not tailored to questions. In effect, this reduces the OpenQA problem to few-passage MRC, imposing a hard limit on the quality of the passages seen by the MRC model. Recent work has sought to address this problem by learning to retrieve passages. For instance, Guu et al. (2020) and Karpukhin et al. (2020) jointly train vector representations of both passages and questions to support similarity-based retrieval. This has led state-of-the-art performance on multiple OpenQAto datasets. However, existing OpenQA retrievers exhibit two major limitations. First, the representa- tions learned by these models are relatively coarse: They encode each passage into a single high-dimensional vector and estimate relevance via one dot-product. We argue this is not expressive enough for reliably matching complex natural-language questions to their answers. Sec- ond, existing systems present substantial tradeoffs when it comes to supervision: they expect hand-labeled ‘‘positive’’ passages, which may not always exist; or they use simple models like BM25 to sample ‘‘positives’’ and ‘‘negatives’’ for training, which may provide weak positives and unchallenging negatives; or they conduct retrieval within the training loop, which requires frequently re-indexing a large corpus (e.g., tens or hundreds of …

Content truncated to 500 words