Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6570–6580 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6570 Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports Baoyu Jing Zeya Wang Petuum Inc., USAEric Xing {baoyu.jing, zeya.wang, eric.xing}@petuum.com Abstract Chest X-Ray (CXR) images are commonly used for clinical screening and diagnosis. Au- tomatically writing reports for these images can considerably lighten the workload of ra- diologists for summarizing descriptive find- ings and conclusive impressions. The com- plex structures between and within sections of the reports pose a great challenge to the auto- matic report generation. Specifically, the sec- tion Impression is a diagnostic summarization over the section Findings; and the appearance of normality dominates each section over that of abnormality. Existing studies rarely ex- plore and consider this fundamental structure information. In this work, we propose a novel framework which exploits the structure infor- mation between and within report sections for generating CXR imaging reports. First, we propose a two-stage strategy that explicitly models the relationship between Findings and Impression. Second, we design a novel co- operative multi-agent system that implicitly captures the imbalanced distribution between abnormality and normality. Experiments on two CXR report datasets show that our method achieves state-of-the-art performance in terms of various evaluation metrics. Our results ex- pose that the proposed approach is able to gen- erate high-quality medical reports through in- tegrating the structure information. 1 Introduction Chest X-Ray (CXR) image report generation aims to automatically generate detailed findings and diagnoses for given images, which has attracted growing attention in recent years (Wang et al., 2018a; Jing et al., 2018; Li et al., 2018). This tech- nique can greatly reduce the workload of radiolo- gists for interpreting CXR images and writing cor- responding reports. In spite of the progress made in this area, it is still challenging for computers Findings: The cardiac silhouette is enlarged and has a globular appearance. Mild bibasilar dependent atelectasis. No pneumothorax or large pleural effusion. No acute bone abnormality. Impression: Cardiomegaly with globular appearance of the cardiac silhouette. Considerations would include pericardial effusion or dilated cardiomyopathy. Figure 1: An example of chest X-ray image along with its report. In the report, the Findings section records de- tailed descriptions for normal and abnormal findings; the Impression section provides a diagnostic conclu- sion. The underlined sentence is an abnormal finding. to accurately write reports. Besides the difficul- ties in detecting lesions from images, the complex structure of textual reports can prevent the success of automatic report generation. As shown in Fig- ure 1, the report for a CXR image usually com- prises two major sections: Findings and Impres- sion. Findings section records detailed descrip- tions about normal and abnormal findings, such as lesions (e.g. increased lung marking). Impression section concludes diseases (e.g. pneumonia) from Findings and forms a diagnostic conclusion, con- sisting of abnormal and normal conclusions. Existing methods (Wang et al., 2018a; Jing et al., 2018; Li …

Content truncated to 500 words