Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1954–1960 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1954 Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays Jianmo Ni∗, Chun-Nan Hsu, Amilcare Gentili, Julian McAuley University of California, San Diego {jin018,chunnan,agentili,jmcauley}@ucsd.edu Abstract Automatic medical image report generation has drawn growing attention due to its poten- tial to alleviate radiologists’ workload. Ex- isting work on report generation often trains encoder-decoder networks to generate com- plete reports. However, such models are af- fected by data bias (e.g. label imbalance) and face common issues inherent in text generation models (e.g. repetition). In this work, we fo- cus on reporting abnormal findings on radiol- ogy images; instead of training on complete ra- diology reports, we propose a method to iden- tify abnormal findings from the reports in addi- tion to grouping them with unsupervised clus- tering and minimal rules. We formulate the task as cross-modal retrieval and propose Con- ditional Visual-Semantic Embeddings to align images and fine-grained abnormal findings in a joint embedding space. We demonstrate that our method is able to retrieve abnormal find- ings and outperforms existing generation mod- els on both clinical correctness and text gener- ation metrics. 1 Introduction Understanding abnormal findings on radiographs (e.g. chest X-Rays) is a crucial task for radiologists. There has been growing interest in automatic radi- ology report generation to alleviate the workload of radiologists and improve patient care. Following the success of neural network models in image- to-text generation tasks (e.g. image captioning), re- searchers have trained CNN-RNN encoder-decoder networks to generate reports given radiology im- ages (Shin et al., 2016; Kougia et al., 2019). Although such models are able to generate fluent reports, the generation quality is often limited by biases introduced from training data or the train- ing process. Figure 1 shows an example of chest ∗ Now at Google X-rays (CXRs) and the associated reports from a public dataset (Johnson et al., 2019), along with the outputs generated by different models.1 One is- sue is that models trained on complete reports tend to generate normal findings as they dominate the dataset (Harzig et al., 2019); another issue is that such generation models struggle to generate long and diverse reports as in other natural language generation (NLG) tasks (Boag et al., 2019). In this work, we focus on reporting abnormal findings on radiology images which are of higher importance to radiologists. To address issues of data bias, we propose a method to identify abnor- mal findings from existing reports and further use K-Means plus minimal mutual exclusivity rules to group these abnormal findings, which reduces the substantial burden of curating templates of abnor- mal findings. Given the fact that radiology reports are highly similar and have a limited vocabulary (Gabriel et al., 2018), we propose a cross-modal retrieval method to capture relevant abnormal find- ings from radiology images. Our contributions are summarized as: • We learn conditional visual-semantic embed- dings on radiology images and reports, which can …

Content truncated to 500 words