SemEval-2015 Task 14: Analysis of Clinical Text

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 303–310, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 14: Analysis of Clinical Text No´emie Elhadad♣, Sameer Pradhan†, Sharon Lipsky Gorman♣, Suresh Manandhar♦, Wendy Chapman♠, Guergana Savova† ♣ Columbia University, USA † Boston Children’s Hospital, USA ♦ University of York, UK ♠ University of Utah, USA noemie.elhadad@columbia.edu, guergana.savova@childrens.harvard.edu Abstract We describe two tasks—named entity recog- nition (Task 1) and template slot filling (Task 2)—for clinical texts. The tasks leverage an- notations from the ShARe corpus, which con- sists of clinical notes with annotated men- tions disorders, along with their normaliza- tion to a medical terminology and eight addi- tional attributes. The purpose of these tasks was to identify advances in clinical named en- tity recognition and establish the state of the art in disorder template slot filling. Task 2 consisted of two subtasks: template slot fill- ing given gold-standard disorder spans (Task 2a) and end-to-end disorder span identifica- tion together with template slot filling (Task 2b). For Task 1 (disorder span detection and normalization), 16 teams participated. The best system yielded a strict F1-score of 75.7, with a precision of 78.3 and recall of 73.2. For Task 2a (template slot filling given gold- standard disorder spans), six teams partici- pated. The best system yielded a combined overall weighted accuracy for slot filling of 88.6. For Task 2b (disorder recognition and template slot filling), nine teams participated. The best system yielded a combined relaxed F (for span detection) and overall weighted ac- curacy of 80.8. 1 Introduction Patient records are abundant with reports, narratives, discussions, and updates about patients. This un- structured part of the record is dense with mentions of clinical entities, such as conditions, anatomical sites, medications, and procedures. Identifying the different entities discussed in a patient record, their status towards the patient, and how they relate to each other is one of the core tasks of clinical natural language processing. Indeed, with robust systems to extract such mentions, along with their associated attributes in the text (e.g., presence of negation for a given entity mention), several high-level applica- tions can be developed such as information extrac- tion, question answering, and summarization. In biomedicine, there are rich lexicons that can be leveraged for the task of named entity recogni- tion and entity linking or normalization. The Uni- fied Medical Language System (UMLS) represents over 130 lexicons/thesauri with terms from a va- riety of languages. The UMLS Metathesaurus in- tegrates standard resources such as SNOMED-CT, ICD9, and RxNORM that are used worldwide in clinical care, public health, and epidemiology. In addition, the UMLS also provides a semantic net- work in which every concept in the Metathesaurus is represented by its Concept Unique Identifier (CUI) and is semantically typed (Bodenreider and McCray, 2003). The SemEval-2015 Task 14, Analysis of Clinical Text is the newest iteration in a series of community challenges organized around the tasks of named en- tity recognition for clinical texts. In SemEval-2014 …

Content truncated to 500 words