Automatic Grading of Evidence: the 2011 ALTA Shared Task

Automatic Grading of Evidence: the 2011 ALTA Shared Task Diego Molla and Abeed Sarker Centre for Language Technology Macquarie Univeristy Sydney, NSW 2109 {diego.molla-aliod, abeed.sarker}@mq.edu.au Abstract The ALTA shared tasks are programming competitions where all participants attempt to solve the same problem, and the winner is the system with the best results. The 2011 ALTA shared task is the second in the series and it focuses on trying to automatically grade the level of clinical evidence in medical research papers. In this paper we describe the task, present the results of several baselines, and the results of our method. We apply a sequence of high precision machine learning classifiers with varying feature sets for each. In addi- tion to using n-grams, we incorporate domain knowledge by representing specific medical concepts using their semantic categories. We also apply a specialised rule-based approach for automatically identifying the publication types of articles, which is then used as a fea- ture set. Our approach obtains an accuracy of 62.84% which is a significant improvement over the baselines. 1 Introduction An important step for physicians who practise Evi- dence Based Medicine (EBM) is the grading of the quality of the clinical evidence present in the medi- cal literature. Evidence grading is a manual process, and the time required to perform it adds to the al- ready time-consuming nature of EBM practice. It has been shown that EBM practitioners often do not pursue evidence based answers to clinical questions because of the time required (Ely et al., 1999; Ely et al., 2005). Therefore, there is a strong motivation for systems that can automatically appraise the evi- dence present in medical publications and generate evidence grades on a specialised scale. The 2011 ALTA shared task addressed the prob- lem of automatic evidence grading. The goal of the task was to build a system that can predict the grade of evidence given a set of medical publications from which the evidence has been extracted. This is a dif- ficult task, and as we show below, machine learning methods that use simple bag-of-word features do not perform significantly better than a trivial baseline. We attempt to solve the problem using supervised machine learning using features such as abstract and title n-grams and publication types. We employ a set of classifiers that utilise the different feature sets and apply them sequentially to obtain an accuracy value of 62.84%, which is a significant improvement over the baseline and also the best result obtained among all the submissions for the shared task. In the following sections, we provide a brief back- ground of EBM, evidence grading, and related work in this area, followed by a description of our meth- ods and the final results. 2 Evidence Based Medicine and Evidence Grading EBM is the ‘conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients’ (Sackett et al., 1996). Current clinical guidelines urge physicians to practise EBM when providing care for their pa- tients. Good …

Content truncated to 500 words