Reinforced Cross-modal Alignment for Radiology Report Generation

Findings of the Association for Computational Linguistics: ACL 2022, pages 448 - 458 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Reinforced Cross-modal Alignment for Radiology Report Generation Han Qin♠, Yan Song♠† ♠The Chinese University of Hong Kong (Shenzhen) ♠hanqin@link.cuhk.edu.cn ♠songyan@cuhk.edu.cn Abstract Medical images are widely used in clinical decision-making, where writing radiology re- ports is a potential application that can be enhanced by automatic solutions to alleviate physicians’ workload. In general, radiology re- port generation is an image-text task, where cross-modal mappings between images and texts play an important role in generating high-quality reports. Although previous stud- ies attempt to facilitate the alignment via the co-attention mechanism under supervised set- tings, they suffer from lacking valid and ac- curate correspondences due to no annotation of such alignment. In this paper, we propose an approach with reinforcement learning (RL) over a cross-modal memory (CMM) to better align visual and textual features for radiology report generation. In detail, a shared memory is used to record the mappings between visual and textual information, and the proposed rein- forced algorithm is performed to learn the sig- nal from the reports to guide the cross-modal alignment even though such reports are not directly related to how images and texts are mapped. Experimental results on two English radiology report datasets, i.e., IU X-Ray and MIMIC-CXR, show the effectiveness of our approach, where the state-of-the-art results are achieved. We further conduct human evalua- tion and case study which confirm the validity of the reinforced algorithm in our approach.1 1 Introduction Radiology report generation aims to automatically generate a free-text description from a specific clin- ical radiograph (e.g., chest X-ray), which can sig- nificantly alleviate the burden of radiologists and thus improve the quality and standardization of health care. With the advantages of its applica- tions, radiology report generation has become an †Corresponding author. 1Our code for this paper is released at https:// github.com/cuhksz-nlp/R2GenRL. Figure 1: A chest X-ray image with its report, where aligned visual and textual features are linked by colors. interesting research topic attracted in both artificial intelligence and clinical medicine. Recently, to generate more accurate reports, approaches based on deep learning techniques are adapted to this task and have achieved great success (Jing et al., 2018; Li et al., 2018; Liu et al., 2019). To effectively perform radiology report gener- ation, most existing studies adopted conventional encoder-decoder architectures with convolutional neural networks (CNN) as the encoder and recur- rent neural networks (e.g., LSTM (Hochreiter and Schmidhuber, 1997), GRU (Cho et al., 2014)) or non-recurrent neural networks (e.g., Transformer (Vaswani et al., 2017)) as the decoder. Considering that there is alignment between radiographs and their corresponding doctor-written reports (such as the mappings demonstrated in Figure 1 where visual and textual features representing the same content are highlighted in the same color), the abil- ity of a model to learn such alignment is the key to achieve outstanding performance. To model the alignment information, Jing et al. (2018) proposed a co-attention mechanism to explicitly learn …

Content truncated to 500 words