Overview of the ALTA 2012 Shared Task

Overview of the ALTA 2012 Shared Task Iman Amini* David Martinez† Diego Molla‡ *RMIT Dept of Computer Science and NICTA, Australia †NICTA and the University of Melbourne, CIS Department, Australia ‡Department of Computing, Macquarie University, Australia iman.amini@rmit.edu.au david.martinez@nicta.edu.au diego.molla-aliod@mq.edu.au Abstract The ALTA shared task ran for the third time in 2012, with the aim of bringing research students together to work on the same task and data set, and compare their methods in a current research problem. The task was based on a recent study to build classifiers for automatically labeling sen- tences to a pre-defined set of categories, in the do- main of Evidence Based Medicine (EBM). The partaking groups demonstrated strong skills this year, outperforming our proposed benchmark sys- tems. In this overview paper we explain the pro- cess of building the benchmark classifiers and data set, and present the submitted systems and their performance. 1 Introduction Medical research articles are one of the main sources for finding answers to clinical queries, and medical practitioners are advised to base their decisions on the available medical litera- ture. Using the literature for the purpose of medi- cal decision making is known as Evidence Based Medicine (EBM). According to the EBM guidelines, users are suggested to formulate queries which follow structured settings, and one of the most used sys- tems is known as PICO: Population (P) (i.e., par- ticipants in a study); Intervention (I); Comparison (C) (if appropriate); and Outcome (O) (of an Inter- vention). This system allows for a better classifi- cation of articles, and improved search. However curating this kind of information manually is un- feasible, due to the large amount of publications being created on daily basis. The goal of the ALTA 2012 shared task was to build automatic sentence classifiers to map the content of biomedical abstracts into a set of pre-defined categories. The development of this kind of technology would speed up the curation process, and this has been explored in recent work (Chung, 2009; Kim et al., 2011). One of the aims of this task was to determine whether par- ticipants could develop systems that can improve over the state of the art. 2 Dataset Different variations and extensions of the PICO classification have been proposed and the schema used for this competition is PIBOSO (Kim et al., 2011), which removes the Comparison tag, and adds three new tags: Background , Study Design and Other. Thus, the tag-set is defined as follows: • Population: The group of individual persons, objects, or items comprising the study’s sam- ple, or from which the sample was taken for statistical measurement; • Intervention: The act of interfering with a condition to modify it or with a process to change its course (includes prevention); • Background: Material that informs and may place the current study in perspective, e.g. work that preceded the current; information about disease prevalence; etc; • Outcome: The sentence(s) that best sum- marise(s) the consequences of an interven- tion; • Study Design: The type of study …

Content truncated to 500 words