KGVL-BART: Knowledge Graph Augmented Visual Language BART for Radiology Report Generation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3401–3411 May 2-6, 2023 ©2023 Association for Computational Linguistics KGVL-BART: Knowledge Graph Augmented Visual Language BART for Radiology Report Generation Kaveri Kale, Pushpak Bhattacharyya IIT Bombay {kaverikale,pb}@cse.iitb.ac.in Milind Gune Augnito India Pvt Ltd dgune@rediffmail.com Aditya Shetty Breach Candy Hospital, India adityashetty01@gmail.com Rustom Lawyer Augnito India Pvt Ltd rustom@augnito.ai Abstract Timely generation of radiology reports and di- agnoses is a challenge worldwide due to the enormous number of cases and shortage of radiology specialists. In this paper, we pro- pose a Knowledge Graph Augmented Vision Language BART (KGVL-BART) model that takes as input two chest X-ray images- one frontal and the other lateral- along with tags which are diagnostic keywords, and outputs a report with the patient-specific findings. Our system development effort is divided into 3 stages: i) construction of the Chest X-ray KG (referred to as chestX-KG), ii) image feature extraction, and iii) training a KGVL-BART model using the visual, text, and KG data. The dataset we use is the well-known Indi- ana University Chest X-ray reports with the train, validation, and test split of 3025 in- stances, 300 instances, and 500 instances re- spectively. We construct a Chest X-Ray knowl- edge graph from these reports by extracting entity1-relation-entity2 triples; the triples get extracted by a rule-based tool of our own. Con- structed KG is verified by two experienced ra- diologists (with experience of 30 years and 8 years, respectively). We demonstrate that our model- KGVL-BART- outperforms State-of- the-Art transformer-based models on standard NLG scoring metrics. We also include a quali- tative evaluation of our system by experienced radiologist (with experience of 30 years) on the test data, which showed that 73% of the reports generated were fully correct, only 5.5% are completely wrong and 21.5% have impor- tant missing details though overall correct. To the best of our knowledge, ours is the first sys- tem to make use of multi-modality and domain knowledge to generate X-ray reports automati- cally. 1 Introduction Medical imaging techniques are widely used in hospitals across the world. The detailed informa- tion extracted from medical images is crucial for proper diagnosis and treatment. An experienced and skilled radiologist is required to prepare an ac- curate full text diagnostic report. However, due to a lack of experts, many reports contain indecisive findings, forcing patients to undergo further tests involving pathology or other advanced imaging methods. In addition, the time consuming process of full text radiology report generation is one of the biggest challenges. All over the world, the ratio of radiologists to patients is very low. The ratios in the US, China, and India are 1:10,000, 1:14,772, and 1:100,000 respectively (Arora, 2014). Given the huge number of cases and the shortage of radiology experts, timely report generation and diagnosis is a huge challenge worldwide. In the case of a chest ailment, X-rays produce images of chest organs like, lungs, spinal bones, heart, airways, and blood vessels. These images help doctors …

Content truncated to 500 words