Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1500–1519, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1500 CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT Akshay Smit * Stanford University akshaysm@stanford.edu Saahil Jain * Stanford University saahil.jain@stanford.edu Pranav Rajpurkar * Stanford University pranavsr@cs.stanford.edu Anuj Pareek Stanford University Andrew Y. Ng Stanford University Matthew P. Lungren Stanford University Abstract The extraction of labels from radiology text re- ports enables large-scale training of medical imaging models. Existing approaches to re- port labeling typically rely either on sophis- ticated feature engineering based on medical domain knowledge or manual annotations by experts. In this work, we introduce a BERT- based approach to medical image report la- beling that exploits both the scale of avail- able rule-based systems and the quality of ex- pert annotations. We demonstrate superior per- formance of a biomedically pretrained BERT model first trained on annotations of a rule- based labeler and then fine-tuned on a small set of expert annotations augmented with au- tomated backtranslation. We find that our fi- nal model, CheXbert, is able to outperform the previous best rule-based labeler with statistical significance, setting a new SOTA for report la- beling on one of the largest datasets of chest x-rays. 1 Introduction The extraction of labels from radiology text re- ports enables important clinical applications, in- cluding large-scale training of medical imaging models (Wang et al., 2017). Many natural language processing systems have been designed to label reports using sophisticated feature engineering of medical domain knowledge (Pons et al., 2016). On chest x-rays, the most common radiological exam, rule-based methods have been engineered to la- bel some of the largest available datasets (Johnson et al., 2019). While these methods have generated considerable advances, they have been unable to capture the full diversity of complexity, ambiguity and subtlety of natural language in the context of radiology reporting. *Equal contribution Radiology Reports (~200,000) Radiology Reports (~1,000) Automatic Labeler (CheXpert) Radiologist BERT (CheXbert) BERT Supervision Input Backtranslation-Augmented Input Transfer Weights Sample Supervision Figure 1: We introduce a method for radiology re- port labeling, in which a biomedically pretrained BERT model is first trained on annotations of a rule-based la- beler, and then fine-tuned on a small set of expert anno- tations augmented with automated backtranslation. More recently, Transformers have demonstrated success in end-to-end radiology report labeling (Drozdov et al., 2020; Wood et al., 2020). How- ever, these methods have shifted the burden from feature engineering to manual annotation, requir- ing considerable time and expertise for high quality. Moreover, these methods do not take advantage of existing feature-engineered labelers, which repre- sent state-of-the-art on many medical tasks. We introduce a simple method for gaining the benefits of both existing radiology report labelers and expert annotations to achieve highly accurate automated radiology report labeling. This approach begins with a biomedically pretrained BERT model (Devlin et al., 2019; Peng et al., 2019) trained on the outputs of an existing labeler, and performs …

Content truncated to 500 words