Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5030–5043 August 1–6, 2021. ©2021 Association for Computational Linguistics 5030 Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision Si Sun1, Yingzhuo Qian2, Zhenghao Liu2, Chenyan Xiong3, Kaitao Zhang2, Jie Bao1, Zhiyuan Liu2, Paul Bennett3 1Department of Electronic Engineering, Tsinghua University, Beijing, China 2Department of Computer Science and Technology, Tsinghua University, Beijing, China Institute for Artificial Intelligence, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology, China 3Microsoft Research, Redmond, USA {s-sun17, qyz17, liu-zh16, zkt18}@mails.tsinghua.edu.cn {bao, liuzy}@tsinghua.edu.cn {chenyan.xiong, Paul.N.Bennett}@microsoft.com Abstract The effectiveness of Neural Information Re- trieval (Neu-IR) often depends on a large scale of in-domain relevance training signals, which are not always available in real-world ranking scenarios. To democratize the benefits of Neu- IR, this paper presents MetaAdaptRank, a do- main adaptive learning method that general- izes Neu-IR models from label-rich source do- mains to few-shot target domains. Drawing on source-domain massive relevance supervision, MetaAdaptRank contrastively synthesizes a large number of weak supervision signals for target domains and meta-learns to reweight these synthetic “weak” data based on their ben- efits to the target-domain ranking accuracy of Neu-IR models. Experiments on three TREC benchmarks in the web, news, and biomed- ical domains show that MetaAdaptRank sig- nificantly improves the few-shot ranking accu- racy of Neu-IR models. Further analyses in- dicate that MetaAdaptRank thrives from both its contrastive weak data synthesis and meta- reweighted data selection. The code and data of this paper can be obtained from https: //github.com/thunlp/MetaAdaptRank. 1 Introduction Text retrieval aims to rank documents to either di- rectly satisfy users’ search needs or find textual information for later processing components, e.g., question answering (Chen et al., 2017) and fact verification (Liu et al., 2020). Neural information retrieval (Neu-IR) models have recently shown ad- vanced results in many ranking scenarios where massive relevance labels or clickthrough data are available (Mitra et al., 2018; Craswell et al., 2020). The flip side is that the “data-hungry” nature of Neu-IR models yields mixed results in few-shot ranking scenarios that suffer from the shortage of labeled data and implicit user feedback (Lin, 2019; Yang et al., 2019). On ranking benchmarks with only hundreds of labeled queries, there have been debates about whether Neu-IR, even with billions of pre-trained parameters (Zhang et al., 2020a), re- ally outperforms traditional IR techniques such as feature-based models and latent semantic index- ing (Yang et al., 2019; Roberts et al., 2020). In fact, many real-world ranking scenarios are few- shot, e.g., tail web queries that innately lack large supervision (Downey et al., 2007), applications with strong privacy constraints like personal and enterprise search (Chirita et al., 2005; Hawking, 2004), and domains where labeling requires pro- fessional expertise such as biomedical and legal search (Roberts et al., 2020; Arora et al., 2018). To broaden the benefits of Neu-IR to few-shot scenarios, we present an adaptive learning method MetaAdaptRank that meta-learns to adapt …

Content truncated to 500 words