SemEval-2014 Task 7: Analysis of Clinical Text

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 54–62, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 7: Analysis of Clinical Text Sameer Pradhan1, No´emie Elhadad2, Wendy Chapman3, Suresh Manandhar4 and Guergana Savova1 1Harvard University, Boston, MA, 2Columbia University, New York, NY 3University of Utah, Salt Lake City, UT, 4University of York, York, UK {sameer.pradhan,guergana.savova}@childrens.harvard.edu, noemie.elhadad@columbia.edu, wendy.chapman@utah.edu, suresh@cs.york.ac.uk Abstract This paper describes the SemEval-2014, Task 7 on the Analysis of Clinical Text and presents the evaluation results. It fo- cused on two subtasks: (i) identification (Task A) and (ii) normalization (Task B) of diseases and disorders in clinical reports as annotated in the Shared Annotated Re- sources (ShARe)1 corpus. This task was a follow-up to the ShARe/CLEF eHealth 2013 shared task, subtasks 1a and 1b,2 but using a larger test set. A total of 21 teams competed in Task A, and 18 of those also participated in Task B. For Task A, the best system had a strict F1-score of 81.3, with a precision of 84.3 and recall of 78.6. For Task B, the same group had the best strict accuracy of 74.1. The organizers have made the text corpora, annotations, and evaluation tools available for future re- search and development at the shared task website.3 1 Introduction A large amount of very useful information—both for medical researchers and patients—is present in the form of unstructured text within the clin- ical notes and discharge summaries that form a patient’s medical history. Adapting and extend- ing natural language processing (NLP) techniques to mine this information can open doors to bet- ter, novel, clinical studies on one hand, and help patients understand the contents of their clini- cal records on the other. Organization of this 1http://share.healthnlp.org 2https://sites.google.com/site/shareclefehealth/ evaluation 3http://alt.qcri.org/semeval2014/task7/ This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ shared task helps establish state-of-the-art bench- marks and paves the way for further explorations. It tackles two important sub-problems in NLP— named entity recognition and word sense disam- biguation. Neither of these problems are new to NLP. Research in general-domain NLP goes back to about two decades. For an overview of the development in the field through roughly 2009, we refer the refer to Nadeau and Sekine (2007). NLP has also penetrated the field of bimedical informatics and has been particularly focused on biomedical literature for over the past decade. Ad- vances in that sub-field has also been documented in surveys such as one by Leaman and Gonza- lez (2008). Word sense disambiguation also has a long history in the general NLP domain (Nav- igli, 2009). In spite of word sense annotations in the biomedical literature, recent work by Savova et al. (2008) highlights the importance of annotat- ing them in clinical notes. This is true for many other clinical and linguistic phenomena as the var- ious characteristics of the clinical narrative present a unique challenge to NLP. Recently various ini- tiatives have …

Content truncated to 500 words