Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 615–624 November 20–23, 2022. ©2022 Association for Computational Linguistics 615 Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations Francesco Dalla Serra1,2 William Clackett1 Chaoyang Wang1 Hamish MacKinnon1 Fani Deligianni2 Jeffrey Dalton2 Alison Q O’Neil1,3 1Canon Medical Research Europe, Edinburgh, United Kingdom 2University of Glasgow, Glasgow, United Kingdom 3University of Edinburgh, Edinburgh, United Kingdom francesco.dallaserra@mre.medical.canon Abstract Automated reporting has the potential to assist radiologists with the time-consuming proce- dure of generating text radiology reports. Most existing approaches generate the report directly from the radiology image, however we observe that the resulting reports exhibit realistic style but lack clinical accuracy. Therefore, we pro- pose a two-step pipeline that subdivides the problem into factual triple extraction followed by free-text report generation. The first step comprises supervised extraction of clinically relevant structured information from the image, expressed as triples of the form (entity1, rela- tion, entity2). In the second step, these triples are input to condition the generation of the radi- ology report. In particular, we focus our work on Chest X-Ray (CXR) radiology report gen- eration. The proposed framework shows state- of-the-art results on the MIMIC-CXR dataset according to most of the standard text genera- tion metrics that we employ (BLEU, METEOR, ROUGE) and to clinical accuracy metrics (re- call, precision and F1 assessed using the CheX- pert labeler), also giving a 23% reduction in the total number of errors and a 29% reduction in critical clinical errors as assessed by expert human evaluation. In future, this solution can easily integrate more advanced model architec- tures – to both improve the triple extraction and the report generation – and can be applied to other complex image captioning tasks, such as those found in the medical domain. 1 Introduction Chest X-Ray (CXR) studies are among the most frequent radiology studies undertaken in health- care (NHS England and NHS improvement, 2022). Each CXR is accompanied by a text report writ- ten by a radiologist or trained radiographer which describes the findings within the study. Unfortu- nately, CXR reports are subject to delays, often due to institutional factors, which can result in ad- verse patient outcomes (Care Quality Commission, 2018). A possible solution to improve the radiology workflow, and to facilitate timely delivery of accu- rate reports, is to automate the generation of text reports. However, generating clinically accurate radiology reports is a challenging task. The task of generating a textual description for an image is referred to as image captioning, and re- cent methods have often adopted encoder-decoder architectures, in which the image embeddings are computed using Convolutional Neural Networks (CNNs) (e.g., He et al., 2016) and the text is gen- erated using Recurrent Neural Networks (RNNs) (e.g., Hochreiter and Schmidhuber, 1997 and Cho et al., 2014), or, more recently, using Transformer- based architectures (Vaswani et al., 2017). Such architectures have …

Content truncated to 500 words