SemEval-2015 Task 6: Clinical TempEval

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 806–814, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 6: Clinical TempEval Steven Bethard University of Alabama at Birmingham Birmingham, AL 35294, USA bethard@cis.uab.edu Leon Derczynski University of Sheffield Sheffield, S1 4DP, UK leon@dcs.shef.ac.uk Guergana Savova Harvard Medical School Boston, MA 02115, USA Guergana.Savova@ childrens.harvard.edu James Pustejovsky, Marc Verhagen Brandeis University Waltham, MA 02453, USA jamesp@cs.brandeis.edu marc@cs.brandeis.edu Abstract Clinical TempEval 2015 brought the tempo- ral information extraction tasks of past Temp- Eval campaigns to the clinical domain. Nine sub-tasks were included, covering problems in time expression identification, event expression identification and temporal relation identifica- tion. Participant systems were trained and eval- uated on a corpus of clinical notes and pathol- ogy reports from the Mayo Clinic, annotated with an extension of TimeML for the clinical domain. Three teams submitted a total of 13 system runs, with the best systems achieving near-human performance on identifying events and times, but with a large performance gap still remaining for temporal relations. 1 Introduction The TempEval shared tasks have, since 2007, pro- vided a focus for research on temporal information extraction (Verhagen et al., 2007; Verhagen et al., 2010; UzZaman et al., 2013). Participant systems compete to identify critical components of the time- line of a text, including time expressions, event ex- pressions and temporal relations. However, the Temp- Eval campaigns to date have focused primarily on in-document timelines derived from news articles. Clinical TempEval brings these temporal informa- tion extraction tasks to the clinical domain, using clinical notes and pathology reports from the Mayo Clinic. This follows recent interest in temporal infor- mation extraction for the clinical domain, e.g., the i2b2 2012 shared task (Sun et al., 2013), and broad- ens our understanding of the language of time beyond newswire expressions and structure. Clinical TempEval focuses on discrete, well- defined tasks which allow rapid, reliable and repeat- able evaluation. Participating systems are expected to take as input raw text such as: April 23, 2014: The patient did not have any postoperative bleeding so we will re- sume chemotherapy with a larger bolus on Friday even if there is slight nausea. And output annotations over the text that capture the following kinds of information: • April 23, 2014: TIMEX3 – TYPE=DATE • postoperative: TIMEX3 –– TYPE=PREPOSTEXP CONTAINS • bleeding: EVENT – POLARITY=NEG – BEFORE document creation time • resume: EVENT – TYPE=ASPECTUAL – AFTER document creation time • chemotherapy: EVENT – AFTER document creation time • bolus: EVENT – AFTER document creation time • Friday: TIMEX3 –– TYPE=DATE CONTAINS • nausea: EVENT – DEGREE=LITTLE – MODALITY=HYPOTHETICAL – AFTER document creation time 806 That is, the systems should identify the time expres- sions, event expressions, attributes of those expres- sions, and temporal relations between them. 2 Data The Clinical TempEval corpus was based on a set of 600 clinical notes and pathology reports from cancer patients at the Mayo Clinic. These notes were man- ually de-identified by the Mayo …

Content truncated to 500 words