On Developing Resources for Patient-level Information Retrieval

On Developing Resources for Patient-level Information Retrieval Stephen Wu1, Tamara Timmons1, Amy Yates1, Meikun Wang1, Steven Bedrick1, William Hersh1, Hongfang Liu2 1Oregon Health & Science University 2Mayo Clinic Portland, OR Rochester, MN E-mail: wst@ohsu.edu liu.hongfang@mayo.edu Abstract Privacy concerns have often served as an insurmountable barrier for the production of research and resources in clinical information retrieval (IR). We believe that both clinical IR research innovation and legitimate privacy concerns can be served by the creation of intra-institutional, fully protected resources. In this paper, we provide some principles and tools for IR resource-building in the unique problem setting of patient-level IR, following the tradition of the Cranfield paradigm. Keywords: clinical information retrieval, privacy, information retrieval evaluation 1. Introduction Privacy concerns have often served as an insurmountable barrier for the production of research and resources in clinical information retrieval (IR). Restrictions on the use of patients’ private health information are a vastly different resource landscape than the more public traditional domains for IR (e.g., web). We believe that both clinical IR research innovation and legitimate privacy concerns can be served by the creation of intra-institutional, fully protected resources. Thus, we have begun creating such resources at two sites: Oregon Health & Science University (OHSU) and Mayo Clinic. In future work, other institutions may choose to adopt these resource development practices; additionally, access could be provided through a privacy-preserving interface, such as with Evaluation-as-a-service (1). In this paper, we provide some principles and tools for resource-building in the unique problem setting of patient-level IR. We follow the long tradition in IR of test collections and challenge evaluations, specifically structuring our resources for Cranfield-style IR evaluations (2, 3). Cranfield evaluations require (i) a test collection, (ii) a set of test topics, expressed as queries, and (iii) judgments of whether documents are relevant, for each query. Each of these 3 collection components needs to be re-envisioned for its role in patient-level IR. In the remainder of our paper, we will walk through the 3 components and provide principles for their design, highlighting how they differ from a “traditional” test collection. We will also describe the implementation of these principles in our own multi-institutional resource-building project. We also introduce the Patient Relevance Assessment Interface (PRAI), a tool for producing relevance judgments for patient-level IR. 2. Related Work The Pittsburgh NLP Repository, distributed exclusively for the 2011-2012 TREC Medical Records Tracks (4, 5), sought to address this problem by providing a de-identified clinical IR collection usable by the research community (albeit with limited availability) for a patient cohort retrieval task. By design, searching this collection bore the marks of patient confidentiality concerns: retrieve hospital visits – as a stand-in for patients – in response to a query. While this was an important step forward in that it moved beyond document retrieval, it did not (and could not) fully embrace patient-level IR. Other research resources and challenges have taken alternative approaches rather than de-identifying protected health information. The CLEF eHealth tasks since 2013 (6) have included patient data …

Content truncated to 500 words