Learning Cross-Lingual IR from an English Retriever

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4428 - 4436 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning Cross-Lingual IR from an English Retriever Yulong Li†∗, Martin Franz‡∗, Md Arafat Sultan‡∗, Bhavani Iyer‡, Young-Suk Lee‡ and Avirup Sil‡ †IBM Research, ‡IBM Research AI {yulongl, franzm, bsiyer, ysuklee, avi}@us.ibm.com arafat.sultan@ibm.com Abstract We present DR.DECR (Dense Retrieval with Distillation-Enhanced Cross-Lingual Representation), a new cross-lingual infor- mation retrieval (CLIR) system trained using multi-stage knowledge distillation (KD). The teacher of DR.DECR relies on a highly effective but computationally expensive two-stage inference process consisting of query translation and monolingual IR, while the student, DR.DECR, executes a single CLIR step. We teach DR.DECR powerful multilingual representations as well as CLIR by optimizing two corresponding KD ob- jectives. Learning useful representations of non-English text from an English-only re- triever is accomplished through a cross-lingual token alignment algorithm that relies on the representation capabilities of the underlying multilingual encoders. In both in-domain and zero-shot out-of-domain evaluation, DR.DECR demonstrates far superior accuracy over direct fine-tuning with labeled CLIR data. It is also the best single-model retriever on the XOR-TyDi benchmark at the time of this writing. 1 Introduction Multilingual models are critical for the democra- tization of AI. Cross-lingual information retrieval (CLIR) (Braschler et al., 1999; Shakery and Zhai, 2013; Jiang et al., 2020; Asai et al., 2021a), for ex- ample, can find relevant text in a high-resource lan- guage such as English even when the query is posed in a different, possibly low-resource, language. In this work, we develop useful CLIR models for this constrained, yet important, setting where a retrieval corpus is available only in a single high-resource language (English in our experiments). A straightforward solution to this problem could use machine translation (MT) to translate the query into English, and then perform English IR (Asai ∗ Equal contribution. et al., 2021a). While such a two-stage process can produce reasonably accurate predictions, an alter- native end-to-end approach that can tackle the prob- lem purely cross-lingually, i.e., without involving MT for inference, would clearly be more efficient and cost-effective. Pre-trained multilingual masked language models (PLMs) such as multilingual BERT (Devlin et al., 2019) or XLM-RoBERTa (XLM-R) (Conneau et al., 2020) can provide the foundation for such a one-step solution, as simply fine-tuning a PLM with labeled CLIR data would yield a cross- lingual retriever (Asai et al., 2021b). Here we first run an empirical evaluation of these two approaches on a public CLIR benchmark (Asai et al., 2021a), which involves both in-domain and zero-shot out-of-domain tests. We use ColBERT (Khattab and Zaharia, 2020; Khattab et al., 2021)— a state-of-the-art (SOTA) neural IR model that has been shown to outperform other recent methods such as DPR (Karpukhin et al., 2020)—as our IR architecture and XLM-R as the underlying PLM for both methods (§2). Results indicate that the MT- based solution can be vastly more effective than direct EN + CLIR fine-tuning, with observed differ- ences …

Content truncated to 500 words