Quick, get me a Dr. BERT: Automatic Grading of Evidence using Transfer Learning

Quick, get me a Dr. BERT: Automatic Grading of Evidence using Transfer Learning Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra, David Eyers Department of Computer Science University of Otago New Zealand [pradeesh,andrew,veronica,dme]@cs.otago.ac.nz Abstract We describe our methods for automatically grading the level of clinical evidence in med- ical papers, as part of the ALTA 2021 shared task. We use a combination of transfer learn- ing and a hand-crafted, feature-based classi- fier. Our system (“orangutanV3”) obtained an accuracy score of 0.4918, which placed third in the leaderboard. From our failure analysis, we find that our classification techniques do not appropriately handle cases when the con- clusions of across the medical papers are them- selves inconclusive. We believe that this shortcoming can be overcome—thus improving the classification accuracy—by incorporating document simi- larity techniques. 1 Introduction The recent COVID-19 pandemic has once again highlighted the importance of Evidence-Based Medicine (EBM) when deciding the course of treatment (Xu et al., 2020) as social media and television shows are being flooded by so-called experts, who have voiced unproven treatments for COVID-19 such as using hydroxychloroquine (Greenhalgh, 2020; Aquino and Cabrera, 2020). However, the main challenge with EBM is that it is a manual and tedious process and it is very hard for practitioners to keep up with the rise in medical research (Ghosh, 2004; Davies, 2007). The challenges with EBM were no less true in 2011, when the Australasian Language Technol- ogy Association (ALTA) organised a shared task challenge to automatically grade evidence (Molla and Sarker, 2011). The task was to grade evidence based on an EBM framework which is the Strength Of Recommendation Taxonomy (SORT) (Ebell et al., 2004). ALTA decided to revisit the 2011 challenge again this year (2021), motivated by the leaps in Natural Language Processing (NLP) tech- niques that have occurred meanwhile (Torfi et al., 2020). We investigate the following research questions with repsect to this challenge: • RQ1: Can we solely use transformers to ac- curately perform SORT? • RQ2: Can we improve the performance of Transformers by incorporating author and journal features? Our experimental results suggest that these two approaches do not perform well. Our team placed third in the leaderboard with an accuracy score of 0.4918. None of the scores on the leaderboard beat the winning accuracy score in 2011, of 0.6284. This prompted us to perform an in-depth analy- sis of our approach, and how our work can be im- proved in the future to increase the overall accu- racy of classification. 2 Related Work In the medical literature, there are many differ- ent taxonomies that are used in order to rank the grade of a clinical study (Abrams et al., 2007; Guyatt et al., 2004). One of the commonly used taxonomies, due to its simplicity, is SORT (Ebell et al., 2004). SORT has been used in deciding whether to recommend root canal treatments (De- Deus and Canabarro, 2017), sports injury reha- bilitation strategies (Bell et al., 2018; Rodriguez et al., 2019), and in evaluation of cognitive be- …

Content truncated to 500 words