ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8108–8122 July 9-14, 2023 ©2023 Association for Computational Linguistics ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning Wenjun Hou1,2, Kaishuai Xu1∗, Yi Cheng1∗, Wenjie Li1†, Jiang Liu2† 1Department of Computing, The Hong Kong Polytechnic University, HKSAR, China 2Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China houwenjun060@gmail.com {kaishuaii.xu, alyssa.cheng}@connect.polyu.hk cswjli@comp.polyu.edu.hk, liuj@sustech.edu.cn Abstract This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs. One sig- nificant challenge of this task is how to cor- rectly maintain the consistency between the im- ages and the lengthy report. Previous research explored solving this issue through planning- based methods, which generate reports only based on high-level plans. However, these plans usually only contain the major observa- tions from the radiographs (e.g., lung opacity), lacking much necessary information, such as the observation characteristics and preliminary clinical diagnoses. To address this problem, the system should also take the image infor- mation into account together with the textual plan and perform stronger reasoning during the generation process. In this paper, we pro- pose an Observation-guided radiology Report GenerAtioN framework (ORGAN). It first pro- duces an observation plan and then feeds both the plan and radiographs for report generation, where an observation graph and a tree reason- ing mechanism are adopted to precisely enrich the plan information by capturing the multi- formats of each observation. Experimental re- sults demonstrate that our framework outper- forms previous state-of-the-art methods regard- ing text quality and clinical efficacy.1 1 Introduction Radiology reports, which contain the textual de- scription for a set of radiographs, are critical in the process of medical diagnosis and treatment. Nev- ertheless, the interpretation of radiographs is very time-consuming, even for experienced radiologists ∗Equal Contribution. †Corresponding authors. 1Our code is available at https://github.com/ wjhou/ORGan. ② Pneumonia / POS ③ Pleural Effusion / NEG ④ Cardiomegaly / NEG ⑤ Support Devices / POS As compared ……, there is ① [a subtle but new opacity] at the right lung base, …… Given the clinical presentation, ② [pneumonia must be suspected]. ……, extensive overinflation with bronchiectasis but ③ [no pleural effusion] or other parenchymal changes. ④ [Normal size of cardiac silhouette]. Unchanged position of the ⑤ [nasogastric tube]. Stage 1: Observation Planning Stage 2: Report Generation ① Lung Opacity / POS Figure 1: Our proposed framework contains two stages, including the observation planning stage and the report generation stage. Red color denotes positive observa- tions, while Blue color denotes negative observations. (5-10 minutes per image). Due to its large potential to alleviate the strain on the healthcare workforce, automated radiology report generation (Anderson et al., 2018; Rennie et al., 2017) has attracted in- creasing research attention. One significant challenge of this task is how to correctly maintain the consistency between the im- age and the lengthy textual report. Many previ- ous works proposed …

Content truncated to 500 words