Competence-based Multimodal Curriculum Learning for Medical Report Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3001–3012 August 1–6, 2021. ©2021 Association for Computational Linguistics 3001 Competence-based Multimodal Curriculum Learning for Medical Report Generation Fenglin Liu1, Shen Ge2, Xian Wu2∗ 1ADSPLAB, School of ECE, Peking University 2Tencent Medical AI Lab, Beijing, China fenglinliu98@pku.edu.cn; {shenge, kevinxwu}@tencent.com Abstract Medical report generation task, which targets to produce long and coherent descriptions of medical images, has attracted growing re- search interests recently. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the se- rious data bias and 2) the limited medical data. To alleviate the data bias and make best use of available data, we propose a Competence- based Multimodal Curriculum Learning frame- work (CMCL). Specifically, CMCL simulates the learning process of radiologists and op- timizes the model in a step by step manner. Firstly, CMCL estimates the difficulty of each training instance and evaluates the competence of current model; Secondly, CMCL selects the most suitable batch of training instances con- sidering current model competence. By iter- ating above two steps, CMCL can gradually improve the model’s performance. The ex- periments on the public IU-Xray and MIMIC- CXR datasets show that CMCL can be incor- porated into existing models to improve their performance. 1 Introduction Medical images, e.g., radiology and pathology im- ages, and their corresponding reports, which de- scribe the observations in details of both normal and abnormal regions, are widely-used for diag- nosis and treatment (Delrue et al., 2011; Goergen et al., 2013). In clinical practice, writing a medi- cal report can be time-consuming and tedious for experienced radiologists, and error-prone for inex- perienced radiologists. Therefore, automatically generating medical reports can assist radiologists in clinical decision-making and emerge as a promi- nent attractive research direction in both artificial ∗Corresponding author. Lungs are clear. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. 1scoliosis. No acute cardiopulmonary abnormality. No focal airspace consolidation. Clear lungs. There is no pneumothorax or pleural effusion. 1Scoliosis is present. No acute bony abnormalities. No pneumothorax or pleural effusion. The heart is normal in size. The lungs are clear. The hilar and mediastinal contours are normal. No evidence of pneumothorax. Ground Truth: Co-Attention: Ours: The heart is enlarged. There is no pneumothorax. No acute bony abnormality. There is a moderate right pleural effusion with associated atelectasis. The left lung is clear. No pneumothorax is seen. The heart and mediastinum are normal. The lungs are clear. 1There is mild blunting of the right costophrenic XXXX. There is no infiltrate, mass or pneumothorax. The right internal jugular catheter has been removed. 1Blunting of right costophrenic. Heart size is normal. No acute bony abnormality. There is no pleural effusion. No visualized pneumothorax. The lungs are clear. Ground Truth: Co-Attention: Ours: Figure 1: Two examples of ground truth reports and reports generated by a state-of-the-art approach Co- Attention …

Content truncated to 500 words