SemEval-2017 Task 12: Clinical TempEval

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 565–572, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 12: Clinical TempEval Steven Bethard University of Arizona Tucson, AZ 85721, USA bethard@email.arizona.edu Guergana Savova Harvard Medical School Boston, MA 02115, USA Guergana.Savova@childrens.harvard.edu Martha Palmer University of Colorado Boulder Boulder, CO 80309 martha.palmer@colorado.edu James Pustejovsky Brandeis University Waltham, MA 02453, USA jamesp@cs.brandeis.edu Abstract Clinical TempEval 2017 aimed to answer the question: how well do systems trained on annotated timelines for one medical con- dition (colon cancer) perform in predict- ing timelines on another medical condition (brain cancer)? Nine sub-tasks were in- cluded, covering problems in time expres- sion identification, event expression identi- fication and temporal relation identification. Participant systems were evaluated on clin- ical and pathology notes from Mayo Clinic cancer patients, annotated with an exten- sion of TimeML for the clinical domain. 11 teams participated in the tasks, with the best systems achieving F1 scores above 0.55 for time expressions, above 0.70 for event expressions, and above 0.30 for tem- poral relations. Most tasks observed about a 20 point drop over Clinical TempEval 2016, where systems were trained and eval- uated on the same domain (colon cancer). 1 Introduction The TempEval shared tasks have, since 2007, pro- vided a focus for research on temporal information extraction (Verhagen et al., 2007, 2010; UzZaman et al., 2013). In recent years the community has moved toward testing such information extraction systems on clinical data, to address a common need of doctors and clinical researchers to search over timelines of clinical events like symptoms, diseases, and procedures. In the Clinical TempEval shared tasks (Bethard et al., 2015, 2016), participant sys- tems have competed to identify critical components of the timeline of a clinical text: time expressions, event expressions, and temporal relations. For ex- ample, Figure 1 shows the annotations that a system is expected to produce when given the text: April 23, 2014: The patient did not have any postoperative bleeding so we’ll re- sume chemotherapy with a larger bolus on Friday even if there is slight nausea. Clinical TempEval 2017 introduced a new aspect to this problem: domain adaptation. Whereas in Clinical TempEval 2015 and 2016, systems were both trained and tested on notes from colon cancer patients, in 2017, systems were trained on colon cancer patients, but tested on brain cancer patients. The diseases, symptoms, procedures, etc. vary widely across these two patient populations, and the doctors treating these different kinds of cancer make a variety of different linguistic choices when discussing such patients. As a result, systems that participated in Clinical TempEval 2017 were faced with a much more challenging task than systems from 2015 or 2016. 2 Data The Clinical TempEval corpus was based on a set of clinical notes and pathology reports from 200 colon cancer patients and 200 brain cancer patients at the Mayo Clinic. These notes were manually de-identified by the Mayo Clinic to replace names, locations, etc. with generic …

Content truncated to 500 words