Beyond [CLS] through Ranking by Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1722–1727, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1722 Beyond [CLS] through Ranking by Generation Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang and Bing Xiang AWS AI New York City, NY {cicnog,xiaofeim,rnallapa,zhiheng,bxiang}@amazon.com Abstract Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document’s language model, were very successful in var- ious IR tasks in the past. However, with the advent of modern deep neural networks, atten- tion has shifted to discriminative ranking func- tions that model the semantic similarity of doc- uments and queries instead. Recently, deep generative models such as GPT2 and BART have been shown to be excellent text gener- ators, but their effectiveness as rankers have not been demonstrated yet. In this work, we revisit the generative framework for informa- tion retrieval and show that our generative ap- proaches are as effective as state-of-the-art se- mantic similarity-based discriminative models for the answer selection task. Additionally, we demonstrate the effectiveness of unlikelihood losses for IR. 1 Introduction Most recent approaches for ranking tasks in Infor- mation Retrieval (IR) such as passage ranking and retrieval of semantically related questions have fo- cused primarily on discriminative methods using neural networks that learn a similarity function to compare questions and candidate answers (Severyn and Moschitti, 2015; dos Santos et al., 2015; Tan et al., 2016; Tay et al., 2017, 2018). On the other hand, classical literature on probabilistic models for IR showed that language modeling, a type of simple generative model, can be effective for docu- ment ranking (Zhai, 2008; Lafferty and Zhai, 2001; Ponte and Croft, 1998). The key idea consists of first training a unique language model lmi for each candidate document di, then using the likelihood of generating the input query using lmi, denoted by P(q|lmi), as the ranking score for document di. Recent advances in neural language models (NLMs) have led to impressive improvements in the quality of automatically generated text (Rad- ford et al., 2019). However, to the best of our knowledge, there is no existing work on exploring the effectiveness of modern generative models such as GPT2, for complex ranking tasks such as answer selection. In this work, we intend to fill this gap by demonstrating that large pretrained generative mod- els can be very effective rankers. Unlike classic LM based approaches for IR that employ separate LMs for each document, our proposed method uses a single global LM that applies to all documents. The global pretrained generator is fine-tuned on the task of query generation conditioned on document content as the context. Additionally, in order to leverage both positive and negative examples, we propose the use of (1) unlikelihood loss on negative examples and (2) ranking loss on the likelihood of positive and negative examples. At inference time, given an input query, our method scores each can- didate document using the likelihood of generating the query …

Content truncated to 500 words