JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5989–6001 October 12–17, 2022. 5989 JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation Jingyi You♡, Dongyuan Li♡, Manabu Okumura♡, Kenji Suzuki♠ Tokyo Institute of Technology ♡{youjy,lidy,oku}@lr.pi.titech.ac.jp ♠suzuki.k.di@m.titech.ac.jp Abstract Automated radiology report generation aims to generate paragraphs that describe fine-grained visual differences among cases, especially those between the normal and the diseased. Existing methods seldom consider the cross- modal alignment between textual and visual features and tend to ignore disease tags as an auxiliary for report generation. To bridge the gap between textual and visual information, in this study, we propose a “Jointly learning framework for automated disease Prediction and radiology report Generation (JPG)” to im- prove the quality of reports through the interac- tion between the main task (report generation) and two auxiliary tasks (feature alignment and disease prediction). The feature alignment and disease prediction help the model learn text- correlated visual features and record diseases as keywords so that it can output high-quality reports. Besides, the improved reports in turn provide additional harder samples for feature alignment and disease prediction to learn more precise visual and textual representations and improve prediction accuracy. All components are jointly trained in a manner that helps im- prove them iteratively and progressively. Exper- imental results demonstrate the effectiveness of JPG on the most commonly used IU X-RAY dataset, showing its superior performance over multiple state-of-the-art image captioning and medical report generation methods with regard to BLEU, METEOR, and ROUGE metrics. 1 Introduction Writing radiology reports and predicting disease labels are two essential procedures in clinical prac- tice. However, manually creating them by radiolo- gists is laborious and time-consuming (Jing et al., 2018; Chen et al., 2021b). Therefore, automated radiology report generation and disease prediction, which aim to generate formal-format descriptive texts (Fig. 1 Findings) and clinical conclusive ter- minologies (Fig. 1 MeSH), have received increas- ing attention recently (Chen et al., 2020; Miura Findings: The$XXXX examination consists of frontal and$lateral$radiographs$ of$the$chest. The cardiac silhouette is not enlarged. Calcified$gran- uloma is$again$seen$in$the$right$upper$lobe.$There$has been$apparent$ interval$increase in low density$convexity$at$the$left$cardiophrenic XXXX.$There$is$no$consolidation,$pleural$effusion$or$pneumothorax.$$ MeSH: Calcified$Granuloma/lung/upper$lobe/right;$ Density/$cardiophrenic angle/left. Figure 1: Chest X-ray images and an accompanying report, including Findings and MeSH labels, from the IU X-RAY dataset. We marked the aligned visual and textual features in different colors for better illustration. et al., 2021; Liu et al., 2021b; Nguyen et al., 2021; Liu et al., 2021c; You et al., 2021a). In particular, they not only improve the efficiency of the entire procedure and liberate people from burdensome workloads, but also maintain the high quality of healthcare. In spite of substantial improvements (Zhang et al., 2020; Wang et al., 2022; Liu et al., 2021a; Shao et al., 2021) have been achieved in the au- tomatic radiology report generation and disease prediction, several challenges remain unsolved. Firstly, following traditional image captioning paradigms (Bhattacharya et al., 2022), current methods mainly adopt a standard encoder-decoder framework with convolutional neural networks (CNNs) encoding radiographs and recurrent neu- ral networks (e.g., LSTM/GRU) …

Content truncated to 500 words