Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval

Proceedings of the 29th International Conference on Computational Linguistics, pages 1071–1082 October 12–17, 2022. 1071 Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval Robert Litschko1 Ivan Vuli´c2 Goran Glavaš1,3 1 Data and Web Science Group, University of Mannheim, Germany 2 Language Technology Lab, University of Cambridge, UK 3 Center for AI and Data Science (CAIDAS), University of Würzburg, Germany Abstract State-of-the-art neural (re)rankers are notori- ously data-hungry which – given the lack of large-scale training data in languages other than English – makes them rarely used in mul- tilingual and cross-lingual retrieval settings. Current approaches therefore commonly trans- fer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders: they fine-tune all parameters of pretrained massively multilin- gual Transformers (MMTs, e.g., multilingual BERT) on English relevance judgments, and then deploy them in the target language(s). In this work, we show that two parameter- efficient approaches to cross-lingual transfer, namely Sparse Fine-Tuning Masks (SFTMs) and Adapters, allow for a more lightweight and more effective zero-shot transfer to multilingual and cross-lingual retrieval tasks. We first train language adapters (or SFTMs) via Masked Lan- guage Modelling and then train retrieval (i.e., reranking) adapters (SFTMs) on top, while keeping all other parameters fixed. At infer- ence, this modular design allows us to compose the ranker by applying the (re)ranking adapter (or SFTM) trained with source language data together with the language adapter (or SFTM) of a target language. We carry out a large scale evaluation on the CLEF-2003 and HC4 bench- marks and additionally, as another contribution, extend the former with queries in three new lan- guages: Kyrgyz, Uyghur and Turkish. The pro- posed parameter-efficient methods outperform standard zero-shot transfer with full MMT fine- tuning, while being more modular and reducing training times. The gains are particularly pro- nounced for low-resource languages, where our approaches also substantially outperform the competitive machine translation-based rankers. 1 Introduction In recent years, neural rankers (Nogueira et al., 2019b; MacAvaney et al., 2019; Khattab and Za- haria, 2020), trained on large-scale datasets (Ba- jaj et al., 2016; Dietz et al., 2017; Craswell et al., 2021), have substantially pushed the performance on various retrieval benchmarks. Since such mod- els are generally too computationally involved (i.e., too slow) for ad-hoc retrieval on large doc- ument collections, they are commonly leveraged as rerankers, i.e., they rerank the output of some fast model (e.g., BM25) that produces the initial ranking. Large-scale datasets for training neural rerankers, however, exist only in English, which impedes their adoption in retrieval scenarios that involve other languages: (a) monolingual retrieval in other languages and (b) cross-lingual informa- tion retrieval (CLIR) in which, for a given query in one language, one needs to determine relevance of documents written in one or more other languages. While CLIR is often instantiated in the form of standalone tasks (e.g., to allow users from differ- ent countries to search over the aggregated global collection of COVID-19 news and findings in their native language (Casacuberta et al., 2021)), …

Content truncated to 500 words