Multi-Task Retrieval for Knowledge-Intensive Tasks

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1098–1111 August 1–6, 2021. ©2021 Association for Computational Linguistics 1098 Multi-Task Retrieval for Knowledge-Intensive Tasks Jean Maillard∗ Vladimir Karpukhin* Fabio Petroni Wen-tau Yih Barlas O˘guz Veselin Stoyanov Gargi Ghosh Facebook AI {jeanm,vladk,fabiopetroni,scottyih,barlaso,ves,gghosh}@fb.com Abstract Retrieving relevant contexts from a large cor- pus is a crucial step for tasks such as open- domain question answering and fact check- ing. Although neural retrieval outperforms tra- ditional methods like tf-idf and BM25, its per- formance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be uni- versal and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only surpasses pre- vious methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing mod- els for downstream tasks and closely match or improve the state of the art on multiple bench- marks. 1 Introduction Knowledge-intensive tasks is the common designa- tion for a class of real-world NLP problems which, because of their nature, require large amounts of knowledge about the world (Petroni et al., 2020). For example, open-domain question answering re- quires producing answers to general factoid ques- tions; fact checking involves determining the vera- city of claims based on a database of trusted evid- ence. Practical solutions to these tasks usually in- volve an efficient retrieval component that, given an input query, selects a limited subset of relevant information from a large knowledge source. Soph- isticated downstream models then consider the in- put only in the context of the retrieved information, and perform the final task.1 ∗Equal Contribution. 1While large pre-trained neural models have been shown to incorporate real-world knowledge in their parameters and thus may skip retrieval (Petroni et al., 2019), they still have limited capacity and suffer from a lack of explainability. The standard retrieval component in many sys- tems (e.g., Thorne et al., 2018; Wang et al., 2018; Chen et al., 2017) has long relied on term-matching methods, such as tf-idf or BM25 (Robertson and Zaragoza, 2009). These methods rely on efficient algorithms and usually perform reasonably well re- gardless of the problem. In contrast, recent neural retrieval models, such as ICT (Lee et al., 2019), DPR (Karpukhin et al., 2020) and RAG (Lewis et al., 2020b) achieve better results by learning directly from task-specific training data and going beyond simple keyword matching. While task spe- cialisation results in improved task performance, researchers have observed that a retriever trained for one specific domain will typically achieve low out-of-domain performance, and even lower per- formance on entirely different tasks (Petroni et al., 2020). This has two implications. First, unlike tf- idf or BM25, neural retrieval models are unsuitable for low data regimes such as few- and zero-shot set- tings. Second, task-specific retrievers complicate practical applications where …

Content truncated to 500 words