Building Evaluation Datasets for Consumer-Oriented Information Retrieval

Building Evaluation Datasets for Consumer-Oriented Information Retrieval Lorraine Goeuriot, Liadh Kelly, Guido Zuccon, Joao Palotti Universit´e Grenoble Alpes, Trinity College Dublin, Queensland University of Technology, Vienna University of Technology France, Ireland, Australia, Austria lorraine.goeuriot@imag.fr, liadh.kelly@tcd.ie, g.zuccon@qut.edu.au, palotti@ifs.tuwien.ac.at Abstract Common people often experience difficulties in accessing relevant, correct, accurate and understandable health information online. Developing search techniques that aid these information needs is challenging. In this paper we present the datasets created by CLEF eHealth Lab from 2013-2015 for evaluation of search solutions to support common people finding health information online. Specifically, the CLEF eHealth information retrieval (IR) task of this Lab has provided the research community with benchmarks for evaluating consumer-centered health information retrieval, thus fostering research and development aimed to address this challenging problem. Given consumer queries, the goal of the task is to retrieve relevant documents from the provided collection of web pages. The shared datasets provide a large health web crawl, queries representing people’s real world information needs, and relevance assessment judgements for the queries. Keywords: consumer health, evaluation campaign, information retrieval 1. Introduction Information retrieval (IR) has evolved as a highly empiri- cal discipline, where evaluation relies on carefully defined protocols and representative test collections. Shared tasks provide test collections to the community, in order to eval- uate, compare and improve information retrieval systems. Shared evaluation test set generation is imperative for pro- gression of the field of information retrieval. In generating test sets of this nature it is important to have robust and representative data. CLEF eHealth is an evaluation campaign that informs the development of approaches to support patients, their next- of-kins, and clinical staff in understanding, accessing and generating health information. The campaign ran yearly since 2013, and each year organizes several tasks related to medical information extraction, management and retrieval. The first CLEFeHealth lab (Suominen et al., 2013) con- tained three tasks: the first one on named entity recognition and/or normalization of disorders (Pradhan et al., 2013); the second one on acronyms/ abbreviations (Mowery et al., 2013) in clinical reports; and the third one on health- focused web information retrieval, supporting laypeople’s information needs stemming from clinical reports (Goeu- riot et al., 2013). The second CLEFeHealth (Kelly et al., 2014) expanded our year-one efforts and again organized three tasks. Specif- ically, the first task aimed to help patients (or their next- of-kin) by addressing visualisation and readability issues related to their hospital discharge documents and related information search on the Internet (Suominen et al., 2014). The second task continued the IE work of the 2013 CLEFe- Health lab, specifically focusing on IE of disorder attributes from clinical text (Mowery et al., 2014). The third task fur- ther extended the 2013 IR task, with a cleaned version of the 2013 document collection being produced and the in- troduction of a new query generation method, as well as multilingual queries (Goeuriot et al., 2014). The 2015 lab was split into two tasks focusing on infor- mation extraction and information retrieval. The IE task introduced …

Content truncated to 500 words