MuSeCLIR: A Multiple Senses and Cross-lingual Information Retrieval Dataset

Proceedings of the 29th International Conference on Computational Linguistics, pages 1128–1135 October 12–17, 2022. 1128 MuSeCLIR: A Multiple Senses and Cross-lingual Information Retrieval dataset Wing Yan Li Julie Weeds David Weir TAG laboratory University of Sussex Brighton, UK {wl238, j.e.weeds, d.j.weir}@sussex.ac.uk Abstract This paper addresses a deficiency in exist- ing cross-lingual information retrieval (CLIR) datasets and provides a robust evaluation of CLIR systems’ disambiguation ability. CLIR is commonly tackled by combining translation and traditional IR. Due to translation ambiguity, the problem of ambiguity is worse in CLIR than in monolingual IR. But existing auto-generated CLIR datasets are dominated by searches for named entity mentions, which does not pro- vide a good measure for disambiguation per- formance, as named entity mentions can of- ten be transliterated across languages and tend not to have multiple translations. Therefore, we introduce a new evaluation dataset (MuSe- CLIR) to address this inadequacy. The dataset focusses on polysemous common nouns with multiple possible translations. MuSeCLIR is constructed from multilingual Wikipedia and supports searches on documents written in Eu- ropean (French, German, Italian) and Asian (Chinese, Japanese) languages. We provide baseline statistical and neural model results on MuSeCLIR which show that MuSeCLIR has a higher requirement on the ability of systems to disambiguate query terms. 1 Introduction Cross-Lingual Information Retrieval (CLIR) is a subfield of Information Retrieval (IR) where the task is to retrieve documents in language Y using queries in language X. Frequently, it is a combined process of translation and conventional IR. For ef- ficiency, it is more common to translate queries from language X to language Y than to translate documents from language Y to language X. Lexical ambiguity is a problem, and disam- biguation found beneficial, in many natural lan- guage processing tasks, including machine trans- lation (Raganato et al., 2019), information extrac- tion (Delli Bovi et al., 2015) and information re- trieval (Blloshmi et al., 2021). This problem is English query Translated query (a) Karen Carroll (judge) KarenCarroll法官 (b) Brian Duffy (chef) BrianDuffy主廚 (c) Larry Andersen LarryAndersen (d) El Cacao, Veraguas ElCacaoVeraguas Table 1: Example queries from BI-139 (Sun and Duh, 2020) translated to Chinese using MUSE (Conneau et al., 2018) dictionaries. exacerbated in CLIR due to translation ambigu- ity (Zhou et al., 2007). For example, letter could mean alphabetic characters (字母) or a message (信). However, in this case, translation ambiguity compounds the problem. 信 can also refer to believ- ing in something and this meaning has nothing to do with letter. Therefore, to increase the precision of the retrieval process, it is crucial to identify the correct translation of the query words in context. Despite its importance, translation ambiguity problem has received relatively little attention by CLIR researchers, and has been overlooked by ex- isting CLIR datasets. Many applications of CLIR, and thus many existing datasets, are dominated by searches for named entity mentions. For example, in the BI-139 English-Chinese dataset (Sun and Duh, 2020), about 73% of the queries contain at least one named entity. These do not tend …

Content truncated to 500 words