MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3876 - 3887 December 7-11, 2022 ©2022 Association for Computational Linguistics MedCLIP: Contrastive Learning from Unpaired Medical Images and Text Zifeng Wang1, Zhenbang Wu1, Dinesh Agarwal1,3, Jimeng Sun1,2 1Department of Computer Science, University of Illinois Urbana-Champaign 2Carle Illinois College of Medicine, University of Illinois Urbana-Champaign 3Adobe {zifengw2, zw12, jimeng}@illinois.edu, diagarwa@adobe.com Abstract Existing vision-text contrastive learning like CLIP (Radford et al., 2021) aims to match the paired image and caption embeddings while pushing others apart, which improves represen- tation transferability and supports zero-shot pre- diction. However, medical image-text datasets are orders of magnitude below the general im- ages and captions from the internet. Moreover, previous methods encounter many false neg- atives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this pa- per, we decouple images and texts for mul- timodal contrastive learning thus scaling the usable training data in a combinatorial magni- tude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero- shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using ≈ 200K data) 1. 1 Introduction Medical images such as X-rays, CTs, and MRIs are commonly used to diagnose, monitor, or treat medical conditions in clinical practice (FDA, 2022). With the rapid growth of medical images and the corresponding reports data, researchers have de- veloped various deep learning models to support clinical decision making (Çallı et al., 2021). Recently, large-scale image-text pre-training, e.g., CLIP (Radford et al., 2021), has achieved considerable successes in computer vision and nat- ural language processing domains. CLIP is trained to predict the correct matching of a batch of images 1Our code is available at https://github.com/ RyanWangZf/MedCLIP. 2050 100 200 570 # of thousands of image-text pairs 40 45 50 55 60 Zero Shot Accuracy (%) (20K, 44.8%) (369K, 42.2%) (191K, 43.3%) 9.6x fewer data MedCLIP ConVIRT GLoRIA Figure 1: Zero-shot performance of MedCLIP, Con- VIRT (Zhang et al., 2020), GLoRIA (Huang et al., 2021) when using different amounts of data for pre-training. ConVIRT and GLoRIA are trained on MIMIC-CXR (369K) and CheXpert (191K) dataset, respectively. Our method yields superior ACC than GLoRIA using near 1/10 of pre-training data. and text training examples. The joint-training of im- age and text representations on large-scale image- text pairs generates transferable representations and supports flexible downstream tasks. Inspired by success of CLIP, we believe the knowledge jointly learned from medical images and reports should be helpful for downstream clinical tasks. However, adopting vision-text pre-training on medical domain is a non-trivial task due to (1) CLIP’s (Radford et al., 2021) data-hungry nature: CLIP is trained on a dataset of 400M image-text pairs collected from the internet, while the total …

Content truncated to 500 words