Generating Radiology Reports via Memory-driven Transformer

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1439–1449, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1439 Generating Radiology Reports via Memory-driven Transformer Zhihong Chen♠♥, Yan Song♠♥†, Tsung-Hui Chang♠♥, Xiang Wan♥ ♠The Chinese University of Hong Kong (Shenzhen) ♥Shenzhen Research Institute of Big Data ♠zhihongchen@link.cuhk.edu.cn ♠{songyan, changtsunghui}@cuhk.edu.cn ♥wanxiang@sribd.cn Abstract Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radi- ologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an es- sential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memory- driven Transformer, where a relational mem- ory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two pre- vailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed ap- proach outperforms previous models with re- spect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our ap- proach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings.1 1 Introduction Radiology report generation, which aims to au- tomatically generate a free-text description for a clinical radiograph (e.g., chest X-ray), has emerged as a prominent attractive research direction in both artificial intelligence and clinical medicine. It can greatly expedite the automation of workflows and improve the quality and standardization of health care. Recently, there are many methods proposed †Corresponding author. 1Our code and the best performing models are released at https://github.com/cuhksz-nlp/R2Gen. Findings The lungs are clear bilaterally. Specifically, no evidence of focal consolidation, or ple- ural effusion. Minimal right basilar subse- gmental atelectasis noted. Cardio medias- tinal silhouette is unremarkable. Tortuosity of the thoracic aorta noted. Impression No acute cardiopulmonary abnormality. Figure 1: An example chest X-ray image and its report including findings and impression. in this area (Jing et al., 2018; Li et al., 2018; John- son et al., 2019; Liu et al., 2019; Jing et al., 2019). Practically, a significant challenge of radiology report generation is that radiology reports are long narratives consisting of multiple sentences. As il- lustrated by Figure 1, a radiology report generally consists of a section of findings which describes medical observations, including both normal and abnormal features, as well as an impression or con- cluding remark summarizing the most prominent observations. Therefore, applying conventional im- age captioning approaches (Vinyals et al., 2015; Anderson et al., 2018) may be insufficient for ra- diology report generation, as such approaches are designed to briefly describe visual scenes with short sentences. The ability to provide accurate clinical descriptions for a radiograph is of the highest pri- ority, which places a …

Content truncated to 500 words