PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain?

Findings of the Association for Computational Linguistics: EACL 2023, pages 1181–1193 May 2-6, 2023 ©2023 Association for Computational Linguistics PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain? Sedigheh Eslami, Christoph Meinel, Gerard de Melo Hasso Plattner Institute / University of Potsdam {sedigheh.eslami, christoph.meinel, gerard.demelo}@hpi.de Abstract Contrastive Language–Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image–text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. In this work, we evaluate the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). We present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments conducted on two MedVQA benchmark datasets illustrate that PubMed- CLIP achieves superior results improving the overall accuracy up to 3% in compari- son to the state-of-the-art Model-Agnostic Meta-Learning (MAML) networks pre-trained only on visual data. The PubMedCLIP model with different back-ends, the source code for pre-training them and reproducing our MedVQA pipeline is publicly available at https://github.com/sarahESL/PubMedCLIP. 1 Introduction Medical visual question answering (MedVQA) seeks answers to natural language questions about a given medical image. The development of Med- VQA has considerable potential to benefit health- care systems, as it may aid clinicians in interpreting medical images and obtaining more accurate diag- noses by consulting a second opinion. Thus, it has become a very active area of research, with compet- itive benchmarks and yearly competitions (Abacha et al., 2021). Yet, visual question answering in the medical domain in particular remains non-trivial, as we suffer from a general lack of large balanced training data, in part due to privacy concerns. To solve the multimodal task of MedVQA, a system must understand both medical images and textual questions and infer the associations between them sufficiently well to produce a correct answer (An- tol et al., 2015). Thus, the success of these solu- tions is tied to the effectiveness of their visual and question encoders. Current approaches for Med- VQA adopt deep artificial neural network encoders to interpret the image and the question. Previous studies in MedVQA (Nguyen et al., 2019; Zhan et al., 2020; Pan et al., 2021; Gong et al., 2022) commonly exploit the Mixture of Enhanced Visual Features (MEVF) model (Nguyen et al., 2019) as their visual encoder to overcome data limitations. However, MEVF is custom-tailored for the par- ticular challenges encountered in the VQA-RAD (Lau et al., 2018) dataset, i.e., specifically designed for the organs present in this dataset, limiting its generalizability to other settings. In non-medical settings, recent work (Su et al., 2019; Zhang et al., 2020; Cho et al., 2021; Wang et al., 2021; Radford et al., 2021; Yu et al., 2022) has shown improvements of visual encoders when learning from multimodal image–text pairs in com- parison to learning from just visual images. Among these approaches, the contrastive pre-training of language–image data in OpenAI’s CLIP (Radford et al., 2021) has been particularly prominent. CLIP is trained …

Content truncated to 500 words