Contrastive Fine-tuning Improves Robustness for Neural Rankers

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 570–582 August 1–6, 2021. ©2021 Association for Computational Linguistics 570 Contrastive Fine-tuning Improves Robustness for Neural Rankers Xiaofei Ma Cicero Nogueira dos Santos AWS AI Labs {xiaofeim, cicnog, anarnld}@amazon.com Andrew O. Arnold Abstract The performance of state-of-the-art neural rankers can deteriorate substantially when ex- posed to noisy inputs or applied to a new domain. In this paper, we present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. Specifically, a contrastive loss that compares data points in the representation space is com- bined with the standard ranking loss during fine-tuning. We use relevance labels to de- note similar/dissimilar pairs, which allows the model to learn the underlying matching se- mantics across different query-document pairs and leads to improved robustness. In experi- ments with four passage ranking datasets, the proposed contrastive fine-tuning method ob- tains improvements on robustness to query re- formulations, noise perturbations, and zero- shot transfer for both BERT and BART-based rankers. Additionally, our experiments show that contrastive fine-tuning outperforms data augmentation for robustifying neural rankers. 1 Introduction Recent advances in neural language modeling have shifted the paradigm of natural language processing (NLP) towards a two-stage process: pre-training on a large amount of data with self-supervised tasks followed by fine-tuning on the target datasets with task-specific loss functions. Current state-of-the- art neural rankers for information retrieval fine- tune pre-trained language models using ranking losses on datasets containing examples of posi- tive and negative query-document pairs. While usually achieving good performance on in-domain test sets, neural rankers trained on large datasets can still exhibit poor transferability when tested in new domains, and suffer from robustness problems when exposed to various types of perturbations. For example, a neural ranker trained on a dataset with mostly natural language queries can perform badly when tested on keyword queries which are very common in information retrieval (Bhatia et al., 2020). A considerable number of previous works have focused on domain adaptation to improve model 's overall transferability. While domain adaptation approaches can help to address the out-of-domain robustness problem (Pan and Yang, 2010; Zhang et al., 2019; Ma et al., 2019), they rely on the avail- ability of either labeled data or at least a target corpus which is usually not available at training time for a neural ranking model deployed in the wild. The vulnerability of deep NLP models to var- ious forms of adversarial attacks such as word- importance-based replacement (Jin et al., 2020), human-curated minimal perturbations (Khashabi et al., 2020), misspelling (Sun et al., 2020), gram- matical errors (Yin et al., 2020), rule-based per- turbations (Si et al., 2020; Ribeiro et al., 2018) is well-documented in the literature (Emma Zhang et al., 2019). While various methods have been proposed to remediate model robustness issues in NLP, most of them are either task-specific (Shah et al., 2019; Zhou et al., 2020; Gan and Ng, 2020; Wang and Bansal, 2018), requiring auxiliary tasks …

Content truncated to 500 words