FAA: Fine-grained Attention Alignment for Cascade Document Ranking

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1688–1700 July 9-14, 2023 ©2023 Association for Computational Linguistics FAA: Fine-grained Attention Alignment for Cascade Document Ranking Zhen Li1, Chongyang Tao2, Jiazhan Feng1, Tao Shen3, Dongyan Zhao1,4∗ Xiubo Geng2, Daxin Jiang2∗ 1Wangxuan Institute of Computer Technology, Peking University 2Microsoft Corporation 3FEIT, University of Technology Sydney 4State Key Laboratory of Media Convergence Production Technology and Systems 1{lizhen63,fengjiazhan,zhaody}@pku.edu.cn 2{chotao,xigeng,djiang}@microsoft.com 3shentao@uts.edu.cn Abstract Document ranking aims at sorting a collection of documents with their relevance to a query. Contemporary methods explore more efficient transformers or divide long documents into pas- sages to handle the long input. However, inten- sive query-irrelevant content may lead to harm- ful distraction and high query latency. Some recent works further propose cascade document ranking models that extract relevant passages with an efficient selector before ranking, how- ever, their selection and ranking modules are almost independently optimized and deployed, leading to selecting error reinforcement and sub-optimal performance. In fact, the docu- ment ranker can provide fine-grained supervi- sion to make the selector more generalizable and compatible, and the selector built upon a different structure can offer a distinct perspec- tive to assist in document ranking. Inspired by this, we propose a fine-grained attention align- ment approach to jointly optimize a cascade document ranking model. Specifically, we uti- lize the attention activations over the passages from the ranker as fine-grained attention feed- back to optimize the selector. Meanwhile, we fuse the relevance scores from the passage se- lector into the ranker to assist in calculating the cooperative matching representation. Experi- ments on MS MARCO and TREC DL demon- strate the effectiveness of our method. 1 Introduction Document ranking aims at ranking the candidate documents according to their relevance to an input query, and it has been widely applied in many nat- ural language processing (NLP) and information retrieval tasks, such as search engines (Hofstätter et al., 2021) and question answering (Chen and Yih, 2020). Due to the powerful representation ability of large-scale pre-trained language mod- els (PLMs) (e.g., BERT (Devlin et al., 2019) and ∗Corresponding author. 𝑝!: … Spiders are animals that have 8 legs and use their fangs to inject venom into other animals and sometimes humans … 𝑝": … Most species trap small insects and other spiders in their webs and eat them. A few large species of spiders prey … 𝑝#: … I caught him and now he lives in a small fish tank, I feed him 2 flies once a day.. He is a fast hunter… 𝑝$: … Thank you so much!Leave a Reply Your email address will not be published … Query: do spiders eat other animals ? … Passages of document Figure 1: The case of scope hypothesis. In this example, p2 is strongly relevant to the query, and p3 is weakly relevant where other passages focus on other topics different from query. RoBERTa (Liu et al., 2019)) that have achieved im- pressive performance in a large number …

Content truncated to 500 words