An Ensemble Model for Automatic Grading of Evidence

An Ensemble Model for Automatic Grading of Evidence Yuting Guo and Yao Ge Computer Science Emory University Atlanta GA 30322, USA yuting.guo@emory.edu yao.ge@emory.edu Ruqi Liao Industrial and System Engineering Georgia Institute of Technology Atlanta GA 30332, USA rliao34@gatech.edu Abeed Sarker Biomedical Informatics Emory University Atlanta GA 30322, USA abeed@dbmi.emory.edu Abstract This paper describes our approach for the auto- matic grading of evidence task from the Aus- tralasian Language Technology Association (ALTA) Shared Task 2021. We developed two classification models with SVM and RoBERTa and applied an ensemble technique to combine the grades from different classifiers. Our results showed that the SVM model achieved compa- rable results to the RoBERTa model, and the ensemble system outperformed the individual models on this task. Our system achieved the first place among five teams and obtained 3.3% higher accuracy than the second place. 1 Introduction Evidence-based medicine (EBM) requires the mak- ing of clinical decisions using the current best exter- nal evidence rather than solely relying on clinical experience and pathophysiologic rationale (Sackett et al., 1996). To adhere to EBM best practice, prac- titioners need to identify the best quality evidence associated with a clinical query. To grade the qual- ity of evidence, Ebell et al. (2004) proposed the Strength of Recommendation Taxonomy (SORT). SORT has a three-levels for rating—A (strong), B (moderate), and C (weak), where A-level is based on high-quality studies with consistent results; B- level is based on high-quality studies with inconsis- tent results or some limitations; C-level is based on the studies with severe limitations. It is a straight- forward grading system that allows clinical experts to rate individual studies or bodies of evidence based on quantity, quality, and consistency. To address the challenging problem of auto- matically grading the quality of evidence, the Australasian Language Technology Association (ALTA) Shared Task 2021 organized a competi- tion. The participants were required to develop a system to predict the grade of evidence given multiple related medical publications. Our team trained several supervised classifiers to address the problem. Our approach included traditional super- vised classification models such as support vec- tor machines (SVM) (Cortes and Vapnik, 1995), neural network models using pretrained models (RoBERTa) (Liu et al., 2019), and an innovative ensemble system which combines the predictions of multiple classifiers. Our results showed that the SVM model achieved comparable results to the RoBERTa model, and the ensemble system outper- formed the individual models on this task. The ensemble model combines the prediction from mul- tiple classifiers in a unique manner: grades (A, B or C) predicted by each classifier is first converted into a continuous number, and then all the numbers are added for each instance. Using the training data, the best separations for the numeric totals are computed. These numeric boundaries are then used to convert continuous scores in the test set to discrete evidence grades. Our system achieved the first place among five teams and obtained 3.3% higher accuracy than the second place. 2 Related Work ALTA Shared Task 2021 is …

Content truncated to 500 words