Adapting Naturally Occurring Test Suites for Evaluation of Clinical Question Answering

Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 21–22, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Adapting naturally occurring test suites for evaluation of clinical question answering Dina Demner-Fushman Lister Hill National Center for Biomedical Communications, National Library of Medicine, NIH, Bethesda, MD 20894, USA ddemner@mail.nih.gov Abstract This paper describes the structure of a test suite for evaluation of clinical question an- swering systems; presents several manually compiled resources found useful for test suite generation; and describes the adaptation of these resources for evaluation of a clinical question answering system. 1 Introduction The community-wide interest in rapid development in many areas of natural language processing and in- formation retrieval resulted in creation of reusable test collections in large-scale evaluations such as the Text REtrieval Conference (TREC)1. Researchers in more specific areas, for which no TREC or other col- lections are available, have to create or find suitable test collections to evaluate their systems. For example, Cramer et al. (2006) recruited vol- unteers and quickly gathered a sizeable corpus of question-answer pairs for evaluation of German open-domain question answering systems. This was achieved through a Web-based tool that allowed marking up “interesting” passages in Wikipedia ar- ticles and then asking questions about the content of those passages. This appealing approach can not easily be applied in the domain of clinical ques- tion answering because the quality of the questions and answers as well as the answer completeness are paramount. A test suite for evaluation of clini- cal question answering systems should contain a set 1http://trec.nist.gov/ of real-life questions asked by clinicians and high- quality answers compiled by experts. The answers should be presented in the form deemed useful by clinicians. One of the benefits of focusing on a specific do- main, such as clinical question answering, is that the user-needs and desirable results are well-studied and their descriptions are readily-available. In the case of clinical question answering, clinicians’ desider- ata are: to see a “bottom-line advice” first, have on-demand access to the context that was used in generation of the advice, and finally have access to the original sources of information (Ely et al., 2005). A fair number of high-quality manually cre- ated collections present answers to clinical questions in this form and could be obtained online. Three par- tially freely-available sources: Family Practitioner Inquiry Network (FPIN)2, Parkhurst Exchange Fo- rum (PE)3, and BMJ Clinical Evidence (BMJ-CE)4 were used to design and develop the presented test suites and evaluation methods. Although there seems to be a distinction between test collections and test suites (Co- hen et al., 2004) (the former defined as “pieces of text” and associated with corpora, the latter, as lists of specially constructed sentences, or sentence sequences, or sentence fragments (Balkan et al., 1994)), evaluation of answers to clinical questions crosses this boundary and requires the availability of carefully generated sentence fragments as well as suitable document collections. 2http://www.primeanswers.org/primeanswers/ 3http://www.parkhurstexchange.com/qa/index.php 4http://www.clinicalevidence.com/ceweb/conditions/index.jsp 21 2 Test suite structure The multi-tiered answer model of the …

Content truncated to 500 words