Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5000–5009 August 1–6, 2021. ©2021 Association for Computational Linguistics 5000 Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation Xingyi Yang∗ UC San Diego x3yang@eng.ucsd.edu Muchao Ye Penn State University muchao@psu.edu Quanzeng You Microsoft Azure Computer Vision quyou@microsoft.com Fenglong Ma† Penn State University fenglong@psu.edu Abstract Medical report generation is one of the most challenging tasks in medical image analysis. Although existing approaches have achieved promising results, they either require a prede- fined template database in order to retrieve sen- tences or ignore the hierarchical nature of med- ical report generation. To address these issues, we propose MedWriter that incorporates a novel hierarchical retrieval mechanism to au- tomatically extract both report and sentence- level templates for clinically accurate report generation. MedWriter first employs the Visual-Language Retrieval (VLR) module to retrieve the most relevant reports for the given images. To guarantee the logical coherence be- tween sentences, the Language-Language Re- trieval (LLR) module is introduced to retrieve relevant sentences based on the previous gen- erated description. At last, a language de- coder fuses image features and features from retrieved reports and sentences to generate meaningful medical reports. We verified the effectiveness of our model by automatic eval- uation and human evaluation on two datasets, i.e., Open-I and MIMIC-CXR. 1 Introduction Medical report generation is the task of generating reports based on medical images, such as radiology and pathology images. Given that this task is time- consuming and cumbersome, researchers endeavor to relieve the burden of physicians by automati- cally generating the findings and descriptions from medical images with machine learning techniques. Existing studies can be roughly divided into two categories, i.e., generation-based and retrieval- based approaches. Generation-based methods, in- cluding LRCN (Donahue et al., 2015), CoAtt (Jing ∗This work was done when Xingyi Yang remotely worked with Dr. Fenglong Ma. † Corresponding author Report-level Retrieval Multi-query Attention Visual Feature Extraction Sentence LSTM Word LSTM Retrieved Reports Sentence-level Retrieval Retrieved Sentences Visual Feature Report Template Sentence Template 𝐕𝐢𝐬𝐮𝐚𝐥 − 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐕𝐋𝐑 𝐌𝐨𝐝𝐮𝐥𝐞 𝐇𝐢𝐞𝐫𝐚𝐫𝐜𝐡𝐢𝐜𝐚𝐥 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐃𝐞𝐜𝐨𝐝𝐞𝐫 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 − 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐋𝐋𝐑 𝐌𝐨𝐝𝐮𝐥𝐞 Figure 1: Overview of the proposed MedWriter. et al., 2018), and MvH+AttL (Yuan et al., 2019), focus on generating image captions with a encoder- decoder model that leverage image features. How- ever, they are unable to produce linguistically di- verse descriptions and depict rare but prominent medical findings. On the other hand, Retrieval- based methods such as HRGR-Agent (Li et al., 2018) and KEPP (Li et al., 2019), pay attention to memorizing templates to generate standardized reports from a predefined retrieval database. How- ever, the quality of generated reports significantly depends on the manually curated template database. Besides, they only use sentence-level templates for the generation but ignore to learn the report-level templates, which prevent them from generating more accurate reports. To address the aforementioned issues, we pro- pose a new framework called MedWriter as shown in Figure 1. …

Content truncated to 500 words