Handling Variance of Pretrained Language Models in Grading Evidence in the Medical Literature

An Approach to the Frugal Use of Human Annotators to Scale up Auto-coding for Text Classification Tasks Li’An Chen1, Hanna Suominen1,3,4 firstname.lastname@anu.edu.au 1 ANU, Centre for the Public Awareness of Science / Canberra, Australia 2 The Australian National University (ANU), School of Computing (SoCo) / Canberra, Australia 3 Data61, Commonwealth Scientific and Industrial Research Organisation / Canberra, Australia 4 University of Turku / Turku, Finland * Address for Correspondence: ANU CPAS, 42A Linnaeus Way, Canberra, ACT 2601, Australia Abstract Human annotation for establishing the training data is often a very costly process in natural language processing (NLP) tasks, which has led to frugal NLP approaches becoming an im- portant research topic. Many research teams struggle to complete projects with limited funding, labor, and computational resources. Driven by the Move-Step analytic framework theorized in the applied linguistics field, our study offers a rigorous approach to the frugal use of two human annotators to scale up auto- coding for text classification tasks. We applied the Linear Support Vector Machine algorithm to text classification of a job ad corpus. Our Cohen’s Kappa for inter-rater agreement and Area Under the Curve (AUC) values reached averages of 0.76 and 0.80, respectively. The calculated time consumption for our human training process was 36 days. The results in- dicated that even the strategic and frugal use of only two human annotators could enable the efficient training of classifiers with reason- ably good performance. This study does not aim to provide generalizability of the results. Rather, it is proposed that the annotation strate- gies arising from this study be considered by our readers only if they are fit for one’s spe- cific research purposes. 1 Introduction In natural language processing (NLP), human an- notation is an indispensable and decisive step. The human annotation process directly influences the quality of the training data in NLP tasks, and con- sequently, it influences the quality of machine- generated results. In this regard, Song et al. (2020) have revealed how significant the risk of reaching an incorrect conclusion could be if the quality of human annotation used for validation cannot be guaranteed. Unfortunately, the science of annota- tion is progressing very slowly (Hovy and Lavid, 2010; Song et al., 2020). In many NLP studies, methodological details concerning the human anno- tation process have not been fully disclosed (Song et al., 2020). Such a lack of disclosure may hinder readers’ judgment of the soundness of human an- notation procedures (Hovy and Lavid, 2010; Song et al., 2020). It is time for NLP researchers to at- tach greater importance to the methodological rigor of human annotation in NLP tasks. Where the funding and labor are limited, institu- tions or researchers might have to turn to the ‘fru- gal’ use of human annotators for text labelling tasks. For instance, Andreotta et al. (2019) acknowledged the limitation of not being able to afford high com- putational and labor costs in their machine learn- ing (ML)-assisted analysis of Tweeter commentary. Johnson et al. (2018) also point out …

Content truncated to 500 words