PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them Patrick Lewis†‡ Yuxiang Wu‡ Linqing Liu‡ Pasquale Minervini‡ Heinrich K¨uttler† Aleksandra Piktus† Pontus Stenetorp‡ Sebastian Riedel†‡ †Facebook AI Research ‡University College London, UK {plewis,hnr,piktus,sriedel}@fb.com {yuxiang.wu,linqing.liu,p.minervini,p.stenetorp}@cs.ucl.ac.uk Abstract Open-domain Question Answering models that directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared with conven- tional models which retrieve and read from text corpora. QA-pair retrievers also offer in- terpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models fall short of the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text cor- pora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically generated QA-pairs. We intro- duce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outper- form comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effec- tiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) while retain- ing high accuracy. Lastly, we demonstrate RePAQ’s strength at selective QA, abstaining from answering when it is likely to be incor- rect. This enables RePAQ to ‘‘back-off’’ to a more expensive state-of-the-art model, lead- ing to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone. 1 Introduction Open-domain QA (ODQA) systems usually have access to a background corpus that can be used to answer questions. Models that explicitly exploit this corpus are commonly referred to as Open- book models (Roberts et al., 2020). They typically index the whole corpus, and then retrieve-and- read documents to answer questions (Voorhees and Harman, 1999; Chen et al., 2017, inter alia). A second class of models, closed-book ques- tion answering (CBQA) models, has recently been proposed. They learn to directly map ques- tions to answers from training question-answer (QA) pairs without access to a background corpus (Roberts et al., 2020; Ye et al., 2021). These mod- els usually take the form of pretrained seq2seq models such as T5 (Raffel et al., 2020) or BART (Lewis et al., 2020a), fine-tuned on QA-pairs. It has recently been shown that current closed-book models mostly memorize training QA-pairs, and can struggle to answer questions that do not overlap with training data (Lewis et al., 2021). Models that explicitly retrieve (training) QA- pairs, rather than memorizing them in parameters, have been shown to perform competitively with CBQA models (Lewis et al., 2021; Xiao et al., 2021). These models have a number of useful pro- perties, such as fast inference, interpretable out- puts (by inspecting retrieved QA-pairs), and the ability to update the model’s knowledge at test time …

Content truncated to 500 words