Cross-modal Contrastive Attention Model for Medical Report Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 2388–2397 October 12–17, 2022. 2388 Cross-modal Contrastive Attention Model for Medical Report Generation Xiao Song 1, Xiaodan Zhang1, Junzhong Ji1,*, Ying Liu2, Pengxu Wei 3 1 Beijing University of Technology 2 Peking University Third Hospital 3 Sun Yat-sen University xiaos@emails.bjut.edu.cn; {zhangxiaodan, jjz01}@bjut.edu.cn; lyyulia@163.com; weipx3@mail.sysu.edu.cn Abstract Medical report automatic generation has gained increasing interest recently as a way to help ra- diologists write reports more efficiently. How- ever, this image-to-text task is rather challeng- ing due to the typical data biases: 1) Normal physiological structures dominate the images, with only tiny abnormalities; 2) Normal de- scriptions accordingly dominate the reports. Existing methods have attempted to solve these problems, but they neglect to exploit useful information from similar historical cases. In this paper, we propose a novel Cross-modal Contrastive Attention (CMCA) model to cap- ture both visual and semantic information from similar cases, with mainly two modules: a Vi- sual Contrastive Attention Module for refining the unique abnormal regions compared to the retrieved case images; a Cross-modal Atten- tion Module for matching the positive semantic information from the case reports. Extensive experiments on two widely-used benchmarks, IU X-Ray and MIMIC-CXR, demonstrate that the proposed model outperforms the state-of- the-art methods on almost all metrics. Further analyses also validate that our proposed model is able to improve the reports with more accu- rate abnormal findings and richer descriptions. 1 Introduction Medical report generation task in practice demands radiologists carefully examine details of images and write corresponding reports, which is time- consuming and technically rigorous. In addition, with the explosion of medical images, generating medical reports has increasingly become a tough burden for radiologists in clinical diagnosis and treatment. Thus, it is extremely desired for auto- matically generating medical reports from medical images, which has also attracted increasing atten- tion especially in the Chest X-ray report generation. Recently, the widely used Encoder-Decoder framework in image captioning task (Karpathy and *Corresponding Author. Ground Truth: There is scarring in the right mid and upper lung zone with surgical clips identified as well. There is no pleural effusion or pneumothorax. The heart is not significantly enlarged. There are atherosclerotic changes of the aorta. changes of the skeletal structures are noted. Baseline: The lungs are clear. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures are normal. CMCA: The lungs are clear. There is no pleural effusion or pneumothorax. The heart is not significantly enlarged. There are calcified mediastinal lymph. There are atherosclerotic changes of the aorta. changes of the skeletal structures are noted. Figure 1: One example of Chest X-ray image with the corresponding ground truth, Baseline (Vaswani et al., 2017) and our model generated reports. The abnormal regions and their corresponding descriptions are marked in same colors, showing serious data biases of this task. Fei-Fei, 2015; You et al., 2016; Vaswani et al., 2017; Anderson et al., 2018) has been success- fully inherited by medical report generation and has …

Content truncated to 500 words