Re2G: Retrieve, Rerank, Generate

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2701 - 2715 July 10-15, 2022 ©2022 Association for Computational Linguistics Re2G: Retrieve, Rerank, Generate Michael Glass1, Gaetano Rossiello1, Md Faisal Mahbub Chowdhury1, Ankita Rajaram Naik1 2, Pengshan Cai1 2, Alfio Gliozzo1 1 IBM Research AI, Yorktown Heights, NY, USA 2 University of Massachusetts Amherst, MA, USA Abstract As demonstrated by GPT-3 and T5, transform- ers grow in capability as parameter spaces be- come larger and larger. However, for tasks that require a large amount of knowledge, non- parametric memory allows models to grow dra- matically with a sub-linear increase in compu- tational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional gen- eration. These models incorporate neural ini- tial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence- to-sequence generation. Our reranking ap- proach also permits merging retrieval results from sources with incomparable scores, en- abling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge dis- tillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, ques- tion answering, fact checking and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source1. 1 Introduction GPT-3 [Brown et al., 2020] and T5 [Raffel et al., 2020] are arguably the most powerful members in a family of deep learning NLP models called transformers. Such models store surprising amount of world knowledge. They have been shown to produce good performance on a range of demand- ing tasks, especially in generating human like texts. However, such large transformers’ capability is tied to the increasingly larger parameter spaces on which they are trained. 1https://github.com/one-qa Recently, there has been work towards trans- formers that make use of non-parametric knowl- edge. REALM (Retrieval Augmented Language Model) [Guu et al., 2020] and RAG (Retrieval Aug- mented Generation) [Lewis et al., 2020b] both use an indexed corpus of passages to support condi- tional generation. By using the corpus as a source of knowledge these models can extend the informa- tion available to the model by tens or even hundreds of gigabytes with a sub-linear scaling in computa- tion cost. These recent advancements, in turn, have been inspired by BART (Bidirectional and Auto- Regressive Transformer) [Lewis et al., 2020a] that combines a Bidirectional Encoder (e.g. BERT [De- vlin et al., 2019]) with an Autoregressive decoder (e.g. GPT [Brown et al., 2020]) into one sequence- to-sequence model. We build on this line of research, pioneered by REALM and RAG, and propose a new ap- proach that we call Re2G (Retrieve, Rerank, Generate), which combines both neural initial re- trieval and …

Content truncated to 500 words