Analyzing Information Retrieval Results With a Focus on Named Entities

Computational Linguistics and Chinese Language Processing Vol. 13, No. 1, March 2008, pp. 121-140 121 © The Association for Computational Linguistics and Chinese Language Processing [Received June 30, 2007; Revised October 8, 2007; Accepted January 12, 2008] Analyzing Information Retrieval Results With a Focus on Named Entities Thomas Mandl∗ and Christa Womser-Hacker* Abstract Experiments carried out within evaluation initiatives for information retrieval have been building a substantial resource for further detailed research. In this study, we present a comprehensive analysis of the data of the Cross Language Evaluation Forum (CLEF) from the years 2000 to 2004. Features of the topics are related to the detailed results of more than 100 runs. The analysis considers the performance of the systems for each individual topic. Named entities in topics revealed to be a major influencing factor on retrieval performance. They lead to a significant improvement of the retrieval quality in general and also for most systems and tasks. This knowledge, gained by data mining on the evaluation results, can be exploited for the improvement of retrieval systems as well as for the design of topics for future CLEF campaigns. Keywords: Cross-Lingual Information Retrieval, Evaluation Issues, Named Entities (NEs) 1. Introduction The Cross Language Evaluation Forum (CLEF) provides a forum for researchers in information retrieval and manages a testbed for mono- and cross-lingual information (CLIR) retrieval systems. CLEF allows the identification of successful approaches, algorithms, and tools in CLIR. Within CLEF, various strategies are employed in order to improve retrieval systems [Braschler and Peters 2004; di Nunzio et al. 2007]. We believe that the effort dedicated to large scale evaluation studies can be exploited beyond the optimization of individual systems. The amount of data created by organizers and participants remains a valuable source of knowledge awaiting exploration. Many lessons can still be learned from past data of evaluation initiatives such as CLEF, TREC [Voorhees and ∗ Information Sci., University of Hildesheim, Marienburger Platz 22, 31141 Hildesheim, Germany. Tel.: +49-5121-883 ext: 837 The author for correspondence is Thomas Mandl. E-mail: mandl@uni-hildesheim.de 122 Thomas Mandl and Christa Womser-Hacker Buckland 2002], INEX [Fuhr 2003], NTCIR [Oyama et al. 2003], or IMIRSEL [Downie 2003]. Ultimately, further criteria and metrics for the evaluation of search and retrieval methods may be found. This could lead to improved algorithms, quality criteria, resources, and tools in cross language information retrieval [Harman 2004; Schneider et al. 2004]. This general research approach is illustrated in Figure 1. Topics are considered an essential component of experiments for information retrieval evaluation [Sparck Jones 1995]. In most evaluations, the variation between topics is larger than the variation between systems. The topic creation for a multilingual test environment requires special care in order to avoid cultural or linguistic bias influencing the semantics of topic formulations [Kluck and Womser-Hacker 2002]. It must be assured that each topic provides equal conditions as starting points for the systems. The question remains whether linguistic aspects randomly appearing within the topics have any influence on the retrieval performance. This is especially important, …

Content truncated to 500 words