CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4160–4170, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4160 CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval Shuo Sun Johns Hopkins University ssun32@jhu.edu Kevin Duh Johns Hopkins University kevinduh@cs.jhu.edu Abstract We present CLIRMatrix, a massively large col- lection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval ex- tracted automatically from Wikipedia. CLIR- Matrix comprises (1) BI-139, a bilingual dataset of queries in one language matched with relevant documents in another language for 139×138=19,182 language pairs, and (2) MULTI-8, a multilingual dataset of queries and documents jointly aligned in 8 different languages. In total, we mined 49 million unique queries and 34 billion (query, doc- ument, label) triplets, making it the largest and most comprehensive CLIR dataset to date. This collection is intended to support research in end-to-end neural information re- trieval and is publicly available at https: //github.com/ssun32/CLIRMatrix. We provide baseline neural model results on BI- 139, and evaluate MULTI-8 in both single- language retrieval and mix-language retrieval settings. 1 Introduction Cross-Lingual Information Retrieval (CLIR) is a retrieval task in which search queries and candi- date documents are written in different languages. CLIR can be very useful in some scenarios. For example, a reporter may want to search foreign- language news to obtain different perspectives for her story; an inventor may explore the patents in another country to understand prior art. Tradition- ally, translation-based approaches are commonly used to tackle the CLIR task (Zhou et al., 2012; Oard, 1998; McCarley, 1999): the query transla- tion approach translates the query into the same language of the documents, whereas the document translation approach translates the document into the same language as the query. Both approaches rely on a machine translation (MT) system or bilin- gual dictionary to map queries and documents to the same language, then employ a monolingual information retrieval (IR) engine to find relevant documents. Recently, the research community has been ac- tively looking at end-to-end solutions that tackle the CLIR task without the need to build MT sys- tems. This line of work builds upon recent ad- vances in Neural Information Retrieval in the mono- lingual setting, c.f. (Mitra and Craswell, 2018; Craswell et al., 2020). There are proposals to di- rectly train end-to-end neural retrieval models on CLIR datasets (Sasaki et al., 2018; Zhang et al., 2019) or MT bitext (Zbib et al., 2019; Jiang et al., 2020). One can also exploit cross-lingual word em- beddings to train a CLIR model on disjoint mono- lingual corpora (Litschko et al., 2018). Despite the growing interest in end-to-end CLIR, the lack of a large-scale, easily-accessible CLIR dataset covering many language directions in high-, mid- and low-resource settings has detrimentally affected the CLIR community’s capability to repli- cate and compare with previously published work. For example, among the widely-used datasets, the CLEF collection (Ferro and Silvello, 2015) covers many languages but is not large enough for training neural models. The …

Content truncated to 500 words