WIKIR: A Python Toolkit for Building a Large-scale Wikipedia-based English Information Retrieval Dataset

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1926–1933 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1926 WIKIR: A Python toolkit for building a large-scale Wikipedia-based English Information Retrieval Dataset Jibril Frej, Didier Schwab, Jean-Pierre Chevallet Univ. Grenoble Alpes, CNRS, Grenoble INP*, LIG, 38000 Grenoble, France * Institute of Engineering Univ. Grenoble Alpes {jibril.frej, didier.schwab, jean-pierre.chevallet}@univ-grenoble-alpes.fr Abstract Over the past years, deep learning methods allowed for new state-of-the-art results in ad-hoc information retrieval. However such methods usually require large amounts of annotated data to be effective. Since most standard ad-hoc information retrieval datasets publicly available for academic research (e.g. Robust04, ClueWeb09) have at most 250 annotated queries, the recent deep learning models for information retrieval perform poorly on these datasets. These models (e.g. DUET, Conv-KNRM) are trained and evaluated on data collected from commercial search engines not publicly available for academic research which is a problem for reproducibility and the advancement of research. In this paper, we propose WIKIR: an open-source toolkit to automatically build large-scale English information retrieval datasets based on Wikipedia. WIKIR is publicly available on GitHub. We also provide wikIR78k and wikIRS78k: two large-scale publicly available datasets that both contain 78,628 queries and 3,060,191 (query, relevant documents) pairs. Keywords: Information Retrieval, Open Source, Dataset, Deep Learning 1. Introduction Deep learning has been shown to be effective in various natural language processing (NLP) tasks such as language modeling, reading comprehension, question answering and natural language understanding (Devlin et al., 2019; Yang et al., 2019b). However, both large and public datasets are key factors for developing effective and reproducible deep learning models. Ad-hoc information retrieval (IR) consists in ranking a set of unstructured documents with respect to a query. Despite the progress in NLP using deep neural networks (DNNs), ad-hoc IR on text documents has not benefited as much as other fields of NLP from DNNs yet (Dehghani et al., 2017). The absence of significant success in ad-hoc IR using deep learning approaches is mainly due to the com- plexity of solving the ranking task using only unlabelled data (Dehghani et al., 2017). Consequently, the availability of large amount of labelled data is crucial to develop effec- tive DNNs for ad-hoc IR. However, as described in Table 1, most of the publicly available English IR datasets only have few labelled data with at most 1,692 labelled queries. Other datasets than the ones presented in Table 1, such as Yahoo! LETOR (Chapelle and Chang, 2011), with more labelled data (≈30k labelled queries) are publicly avail- able. However, only the feature vectors describing query- document pairs are provided. Such datasets are suitable for feature-based learning-to-rank models but not for DNNs that require the original content of queries and documents. Thus, most of the deep learning model for ad-hoc IR that have been proposed recently are developed using one of the following approaches: (1) Using large amounts of data collected from commercial search engines that are not publicly available (Yang et …

Content truncated to 500 words