Auto-Encoding Questions with Retrieval Augmented Decoding for Unsupervised Passage Retrieval and Zero-Shot Question Generation

Proceedings of Recent Advances in Natural Language Processing, pages 1171–1179 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_124 1171 AutoQIR: Auto-Encoding Questions with Retrieval Augmented Decoding for Unsupervised Passage Retrieval and Zero-shot Question Generation Stalin Varanasi1,2 Muhammad Umer Butt1 G¨unter Neumann1,2 1Saarland Informatics Campus, D3.2, Saarland University, Germany 2 German Research Center for Artificial Intelligence (DFKI), Saarbr¨ucken, Germany stalin.varanasi@dfki.de, mubu00001@uni-saarland.de, guenter.neumann@dfki.de Abstract Dense passage retrieval models have be- come state-of-the-art for information retrieval on many Open-domain Question Answering (ODQA) datasets. However, most of these models rely on supervision obtained from the ODQA datasets, which hinders their per- formance in a low-resource setting. Re- cently, retrieval-augmented language models have been proposed to improve both zero-shot and supervised information retrieval. How- ever, these models have pre-training tasks that are agnostic to the target task of passage re- trieval. In this work, we propose Retrieval Aug- mented Auto-encoding of Questions for zero- shot dense information retrieval. Unlike other pre-training methods, our pre-training method is built for target information retrieval, thereby making the pre-training more efficient. Our method consists of a dense IR model for en- coding questions and retrieving documents dur- ing training and a conditional language model that maximizes the question’s likelihood by marginalizing over retrieved documents. As a by-product, we can use this conditional lan- guage model for zero-shot question generation from documents. We show that the IR model obtained through our method improves the cur- rent state-of-the-art of zero-shot dense infor- mation retrieval, and we improve the results even further by training on a synthetic corpus created by zero-shot question generation. 1 Introduction Open Domain Question Answering (ODQA) with dense passage retrieval has been quite successful in recent years. This is primarily because of the availability of large question-answering corpora. However, annotations for the creation of Open- Domain Question Answering (ODQA) datasets consume significant time and effort, although, the indispensable need for labeled data is evident in the decline of cross-domain performance across vari- ous ODQA datasets (Karpukhin et al., 2020) for both information retrieval and question-answering tasks. To this end, in this work, we address the task of Unsupervised Dense Passage Retrieval (UDPR). That is, to be able to retrieve relevant documents without the labels on ground truth question-passage pairs, which reflects a real-world scenario. In this work, we propose Retrieval Augmented Auto-Encoding of Questions (named as AutoQIR1) as a means to obtain similarity between documents and questions to perform zero-shot dense passage retrieval. Our method not only complements the supervised methods but unlike other zero-shot pre- trained models, it also considers a pre-training task that is directly relevant to questions. The following are the contributions of this work: 1. We propose a novel pre-training task for Un- supervised Dense Information Retrieval. 2. We provide a new method for zero-shot ques- tion generation which can be used for data augmentation of Question Answering/ IR Datasets. 3. We provide a way to transfer knowledge from language models to Information Retrieval. 4. Our method surpasses the baseline and is on par with other zero-shot …

Content truncated to 500 words