SemEval-2016 Task 12: Clinical TempEval

SemEval-2016 Task 12: Clinical TempEval Steven Bethard University of Alabama at Birmingham Birmingham, AL 35294, USA bethard@cis.uab.edu Guergana Savova Harvard Medical School Boston, MA 02115, USA Guergana.Savova@childrens.harvard.edu Wei-Te Chen University of Colorado Boulder Boulder, CO 80309 weite.chen@colorado.edu Leon Derczynski University of Sheffield Sheffield, S1 4DP, UK leon.derczynski@sheffield.ac.uk James Pustejovsky Brandeis University Waltham, MA 02453, USA jamesp@cs.brandeis.edu Marc Verhagen Brandeis University Waltham, MA 02453, USA marc@cs.brandeis.edu Abstract Clinical TempEval 2016 evaluated temporal information extraction systems on the clinical domain. Nine sub-tasks were included, cover- ing problems in time expression identification, event expression identification and temporal re- lation identification. Participant systems were trained and evaluated on a corpus of clinical and pathology notes from the Mayo Clinic, an- notated with an extension of TimeML for the clinical domain. 14 teams submitted a total of 40 system runs, with the best systems achieving near-human performance on identifying events and times. On identifying temporal relations, there was a gap between the best systems and human performance, but the gap was less than half the gap of Clinical TempEval 2015. 1 Introduction The TempEval shared tasks have, since 2007, pro- vided a focus for research on temporal information extraction (Verhagen et al., 2007; Verhagen et al., 2010; UzZaman et al., 2013). Participant systems compete to identify critical components of the time- line of a text, including time expressions, event ex- pressions and temporal relations. However, the Temp- Eval campaigns to date have focused primarily on in-document timelines derived from news articles. In recent years, the community has moved toward testing such information extraction systems on clin- ical data (Sun et al., 2013; Bethard et al., 2015) to broaden our understanding of the language of time beyond newswire expressions and structure. Clinical TempEval focuses on discrete, well- defined tasks which allow rapid, reliable and repeat- able evaluation. Participating systems are expected to take as input raw text, for example: April 23, 2014: The patient did not have any postoperative bleeding so we’ll resume chemotherapy with a larger bolus on Friday even if there is slight nausea. The systems are then expected to output annotations over the text, for example, those shown in Figure 1. That is, the systems should identify the time expres- sions, event expressions, attributes of those expres- sions, and temporal relations between them. Clinical TempEval 2016 addressed one of the ma- jor challenges in Clinical TempEval 2015: data dis- tribution. Because Clinical TempEval is based on real patient notes from the Mayo Clinic, participants go through a lengthy authorization process involving a data use agreement and an interview. For Clini- cal TempEval 2016, we streamlined this process and were able to authorize data access for more than twice as many participants as Clinical TempEval 2015. And since all the training and evaluation data distributed for Clinical TempEval 2015 was used as the training data for Clinical TempEval 2016, participants had more than a year to work on their systems. The result was that four times as many teams participated. TIMEX3 CLASS=DATE April 23, …

Content truncated to 500 words