Evaluation of Medical Concept Annotation Systems on Clinical Records

Evaluation of Medical Concept Annotation Systems on Clinical Records Hamed Hassanzadeh Anthony Nguyen Bevan Koopman The Australian e-Health Research Centre, CSIRO, Brisbane, QLD, Australia {hamed.hassanzadeh, anthony.nguyen, bevan.koopman}@csiro.au Abstract Large volumes of electronic health records, including free-text documents, are extensively generated within various sectors of healthcare. Medical concept annotation systems are designed to enrich these documents with key concepts in the domain using reference terminolo- gies. Although there is a wide range of annotation systems, there is a lack of comparative analysis that enables thorough understanding of the effec- tiveness of both the concept extraction and concept recognition components of these systems, especially within the clinical domain. This paper analyses and evaluates four annotation systems (i.e., MetaMap, NCBO annotator, Ontoserver, and QuickUMLS) for the task of ex- tracting medical concepts from clinical free-text documents. Empirical findings have shown that each annotator exhibits various levels of strengths in terms of overall precision or recall. The concept recognition component of each system, however, was found to be highly sensitive to the quality of the text spans output by the concept extraction component of the annotation system. The effects of these components on each other are quantified in such way as to provide evidence for an informed choice of an annotation system as well as avenues for future research. 1 Introduction With the advent of electronic health records, large volumes of mostly free-text clinical documents — discharge summaries, radiology reports, pathol- ogy reports, and patients progress notes — are now present in the health ecosystem. While these documents contain much valuable information, it can only be exploited if effective computational methods of dealing with clinical free-text are de- vised. The goal here is to automatically extract clinical concepts from unstructured clinical docu- ments, thus providing a structured representation that enables fast and effective access and analysis. To facilitate the extraction of clinical concepts from free-text, many automatic systems (known as medical concept annotators) have been devel- oped. These systems analyse natural language and annotate specific spans of text to concepts defined in some external medical terminology/thesaurus. This workflow can be considered as a two-step process of extracting candidate spans of concepts within a given document (known as “concept ex- traction”) and then assigning appropriate concept identifiers to each candidate span based on the de- fined concepts in the domain ontologies (known as “concept recognition”). Such systems are widely used in a variety of e-health settings and are criti- cal for activities such as clinical information anal- ysis and reporting (Zuccon et al., 2013), derivation of phenotypic descriptions (Groza et al., 2013b; Collier et al., 2014) and medical information re- trieval (Zuccon et al., 2012; Koopman, 2014). Although there are a wide range of available an- notation systems, there is a lack of comparative analysis that provides enough evidence for an in- formed decision in choosing the most suitable sys- tem. Many of these system are developed for a specific domain (e.g., medical journal article ab- stracts) and may not be suited to dealing with …

Content truncated to 500 words