Automated Generation of Accurate & Fluent Medical X-ray Reports

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3552–3569 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3552 Automated Generation of Accurate & Fluent Medical X-ray Reports Hoang T.N. Nguyen* University of Alberta Dong Nie* University of North Carolina at Chapel Hill Taivanbat Badamdorj University of Alberta Yujie Liu Guangzhou University of Chinese Medicine Yingying Zhu University of Texas at Arlington Jason Truong University of Alberta Li Cheng University of Alberta Abstract Our paper aims to automate the generation of medical reports from chest X-ray image inputs, a critical yet time-consuming task for radiolo- gists. Existing medical report generation ef- forts emphasize producing human-readable re- ports, yet the generated text may not be well aligned to the clinical facts. Our generated medical reports, on the other hand, are flu- ent and, more importantly, clinically accurate. This is achieved by our fully differentiable and end-to-end paradigm that contains three com- plementary modules: taking the chest X-ray images and clinical history document of pa- tients as inputs, our classification module pro- duces an internal checklist of disease-related topics, referred to as enriched disease embed- ding; the embedding representation is then passed to our transformer-based generator, to produce the medical report; meanwhile, our generator also creates a weighted embedding representation, which is fed to our interpreter to ensure consistency with respect to disease- related topics. Empirical evaluations demon- strate very promising results achieved by our approach on commonly-used metrics concern- ing language fluency and clinical accuracy. Moreover, noticeable performance gains are consistently observed when additional input in- formation is available, such as the clinical doc- ument and extra scans from different views. 1 Introduction Medical reports are the primary medium, through which physicians communicate findings and diag- noses from the medical scans of patients. The pro- cess is usually laborious, where typing out a medi- cal report takes on average five to ten minutes (Jing et al., 2018); it could also be error-prone. This has led to a surging need for automated generation of medical reports, to assist radiologists and physi- cians in making rapid and meaningful diagnoses. * indicates equal contribution. Code is avail- able at https://github.com/ginobilinie/xray_ report_generation Its potential efficiency and benefits could be enor- mous, especially during critical situations such as COVID or a similar pandemic. Clearly a successful medical report generation process is expected to possess two key properties: 1) clinical accuracy, to properly and correctly describe the disease and related symptoms; 2) language fluency, to produce realistic and human-readable text. Fueled by recent progresses in the closely related computer vision problem of image-based caption- ing (Vinyals et al., 2015; Tran et al., 2020), there have been a number of research efforts in medical report generation in recent years (Jing et al., 2018, 2019; Li et al., 2018, 2019; Xue et al., 2018; Yuan et al., 2019; Wang et al., 2018; Yin et al., 2019; Lovelace and Mortazavi, 2020; Srinivasan et al., 2020). These methods often perform reasonably well in addressing the language fluency …

Content truncated to 500 words