Overview of the 2021 ALTA Shared Task: Automatic Grading of Evidence, 10 years later

Overview of the 2021 ALTA Shared Task: Automatic Grading of Evidence, 10 years later Diego Moll´a Department of Computing Macquarie University diego.molla-aliod@mq.edu.au Abstract The 2021 ALTA shared task is the 12th instance of a series of shared tasks organised by ALTA since 2010. Motivated by the advances in ma- chine learning in the last 10 years, this year’s task is a re-visit of the 2011 ALTA shared task. Set within the framework of Evidence Based Medicine (EBM), the goal is to predict the qual- ity of the clinical evidence present in a set of documents. This year’s participant results did not improve over those of participants from 2011. 1 Introduction Evidence Based Medicine (EBM) urges the med- ical practitioner to make use of the best available evidence for making decisions about the care of individual patients (Sackett et al., 1996). However, medical and biomedical research generates such a volume of publications that it is unrealistic for a medical doctor or researcher to be able to read all relevant publications in order to be up to date on the available medical evidence. For example, PubMed currently contains more than 33 million citations for biomedical literature1. A more recent collec- tion, CORD-19, contains over 500,000 publications on topics related to COVID-19, SARS-CoV-2, and related coronaviruses2. An important step for determining the best clini- cal evidence is to grade the quality of the available evidence. To help address this problem, in 2011 the ALTA shared task launched the task of automatic evidence grading (Moll´a and Sarker, 2011). The goal of the task was to build a system that predicts the grade of evidence available in a set of medical publications. Forward 10 years, in 2021, the task has been re-visited. The 2021 task uses the same 1https://pubmed.ncbi.nlm.nih.gov/ 2https://www.kaggle.com/allen-institute-for-ai/CORD- 19-research-challenge training and test data sets as in 2011, and the evalu- ation framework has been re-created as closely as possible to match the 2011 evaluation framework. We wanted to know whether the recent advances in machine learning over the last 10 years lead to an improvement in the accuracy of the automatic grad- ing of evidence predictors. This paper describes the specific set up of the 2021 ALTA shared task, and shows the results of the participating systems. Back in 2011, no participating systems improved on a majority baseline. In 2021, the results of the participating systems appear to improve over the majority baseline, but the difference is not statisti- cally significant. Section 2 gives more details about the automatic grading of evidence task. Section 3 presents related work since 2011. Section 4 details the evaluation framework. Section 5 presents the participating systems and their results, and Sec- tion 6 concludes this paper. 2 Evidence Grading Several taxonomies have been defined to grade the quality of the medical evidence. The Strength of Recommendation Taxonomy (SORT) (Ebell et al., 2004), used in the 2011 ALTA shared task, is one such taxonomy. SORT uses a 3-point scale defined as follows: A Recommendation based on consistent and …

Content truncated to 500 words