Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288–5304 June 6–11, 2021. ©2021 Association for Computational Linguistics 5288 Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P. Langlotz, Dan Jurafsky Stanford University {ysmiura, zyh, ebtsai, langlotz, jurafsky}@stanford.edu Abstract Neural image-to-text radiology report gener- ation systems offer the potential to improve radiology reporting by reducing the repeti- tive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural language genera- tion metrics such as CIDEr or BLEU, still suf- fer from incomplete and inconsistent genera- tions. Here we introduce two new simple re- wards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate radiol- ogy domain entities consistent with the refer- ence, and one that uses natural language in- ference to encourage these entities to be de- scribed in inferentially consistent ways. We combine these with the novel use of an exist- ing semantic equivalence metric (BERTScore). We further propose a report generation sys- tem that optimizes these rewards via reinforce- ment learning. On two open radiology report datasets, our system substantially improved the F1 score of a clinical information extrac- tion performance by +22.1 (∆ + 63.9%). We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines. 1 Introduction An important new application of natural language generation (NLG) is to build assistive systems that take X-ray images of a patient and generate a tex- tual report describing clinical observations in the images (Jing et al., 2018; Li et al., 2018; Liu et al., 2019; Boag et al., 2020; Chen et al., 2020). Figure 1 shows an example of a radiology report generated by such a system. This is a clinically important task, offering the potential to reduce radiologists’ repetitive work and generally improve clinical com- munication (Kahn et al., 2009). Reference Report Large right pleural effusion is unchanged in size. There is associated right basilar atelectasis/scarring, also stable. Healed right rib fractures are noted. On the left, there is persistent apical pleural thickening and apical scarring. Linear opacities projecting over the lower lobe are also compatible with scarring, unchanged. There is no left pleural effusion. There is no pneumothorax. … Medical Images Image Encoder Text Decoder Generated Report … The heart size remains unchanged and is within normal limits. Unchanged appearance of thoracic aorta. The pulmonary vasculature is not congested. Bilateral pleural effusions are again noted and have increased in size on the right than the left. The left-sided pleural effusion has increased in size and is now moderate in size. contradiction Figure 1: A (partial) example of a report generated from our system (with “...” representing abbreviated text). The system encodes images and generates text from that encoded representation. Underlined …

Content truncated to 500 words