CXR Data Annotation and Classification with Pre-trained Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 2801–2811 October 12–17, 2022. 2801 Abstract Clinical data annotation has been one of the major obstacles for applying machine learning approaches in clinical NLP. Open- source tools such as NegBio and CheXpert are usually designed on data from specific institutions, which limit their applications to other institutions due to the differences in writing style, structure, language use as well as label definition. In this paper, we propose a new weak supervision annotation framework with two improvements compared to existing annotation frameworks: 1) we propose to select representative samples for efficient manual annotation; 2) we propose to auto-annotate the remaining samples, both leveraging on a self-trained sentence encoder. This framework also provides a function for identifying inconsistent annotation errors. The utility of our proposed weak supervision annotation framework is applicable to any given data annotation task, and it provides an efficient form of sample selection and data auto-annotation with better classification results for real applications. 1 Introduction Previous work (Wang et al., 2019) has showed clinical text classification can significantly benefit from supervised learning approaches when annotated data is available. However, annotation of clinical data is extremely expensive and time consuming since annotation needs human experts’ domain knowledge. To circumvent this, some rule-based methods have been developed using expert knowledge, e.g., NegBio (Peng et al., 2018) and CheXpert (Irvin et al., 2019) using rule labeler to automatically detect the presence of observations in radiology reports. McDermott et al. (2020) mentioned that CheXpert is computationally slow, and its output is non- differentiable, so they proposed to train a BERT based classier (CheXpert++) based on the output of CheXpert. Likewise, Smit et al. (2020) proposed to combine automatic labelers with expert knowledge by first fine-tuning BERT classifier on output of CheXpert and then on a small set of expert annotations augmented with automated backtranslation. While annotation is data and task specific (Irena et al., 2020), it is further complicated by the differences in writing style, structure, language use such as the vocabularies and phrase variability among different institutes and different countries. For example, phrases “airspace changes” and “infective change(s)” are commonly seen in our local data to describe pneumonia, but rarely seen in MIMIC data (Alistair et al., 2019). In addition, label definitions vary among institutes. For example, CheXpert classifies the sentence “suspicious for pneumonia” as “pneumonia uncertainty” based on their rule definition, while our clinicians/radiologists would consider it as an implication of “pneumonia positive”. These differences limit the application of open-source tools on different data, and further limit the applications of methods (McDermott et al., 2020; Smit et al., 2021) which heavily rely on open- source tools. Some active learning (Chen et al., 2015) and interactive learning methods (Wang et al., 2017) have been commonly used for reducing the experts’ annotation burden. Chen et al. (2015) CXR data annotation and classification with pre-trained language models Nina Zhou1, Ai Ti Aw1, Zhuohan Liu1, Cher Heng Tan2, Yonghan Ting2, Wenxiang Chen2, and Sim Zheng …

Content truncated to 500 words