SLT at SemEval-2023 Task 1: Enhancing Visual Word Sense Disambiguation through Image Text Retrieval using BLIP

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1921–1925 July 13-14, 2023 ©2023 Association for Computational Linguistics SLT at SemEval-2023 Task 1: Enhancing Visual Word Sense Disambiguation through Image Text Retrieval using BLIP Mohammadreza Molavi Department of Computer Engineering Amirkabir University of Technology mmdreza.molavi@aut.ac.ir Hossein Zeinali Department of Computer Engineering Amirkabir University of Technology hzeinali@aut.ac.ir Abstract Based on recent progress in image-text retrieval techniques, this paper presents a fine-tuned model for the Visual Word Sense Disambigua- tion (VWSD) task. The proposed system fine- tunes a pre-trained model using ITC and ITM losses and employs a candidate selection ap- proach for faster inference. The system was trained on the VWSD task dataset and evalu- ated on a separate test set using Mean Recipro- cal Rank (MRR) metric. Additionally, the sys- tem was tested on the provided test set which contained Persian and Italian languages, and the results were evaluated on each language separately. Our proposed system demonstrates the potential of fine-tuning pre-trained models for complex language tasks and provides in- sights for further research in the field of image text retrieval. 1 Introduction Visual word sense disambiguation (VWSD) orga- nized by (Raganato et al., 2023)1 is the task of selecting the correct image from a set of candidate images that corresponds to the intended meaning of a target word, given a limited textual context. SemEval 2023 Task 1 focuses on VWSD in the context of images and textual descriptions. VWSD helps to improve the performance of many natu- ral language processing (NLP) applications that involve both textual and visual information. For example, in image captioning(Hossain et al., 2019), the system needs to identify the correct sense of a word in a given context to generate an accurate and meaningful caption. Similarly, in visual question answering (VQA) (Antol et al., 2015), the system needs to understand the meaning of the question and the visual context to provide a correct answer. Image retrieval aims to retrieve images that are relevant to a given textual query, such as a caption 1https://raganato.github.io/vwsd/ or a keyword. This involves learning a representa- tion of both images and text in a shared embedding space, where the similarity between images and text is determined by their proximity in the em- bedding space. The objective is to locate the most relevant images based on the textual query and present them in a ranked order. (Datta et al., 2008) Vision-language pre-training has emerged as a powerful approach for multimodal understanding in recent years (e.g., CLIP (Radford et al., 2021), ALBEF (Li et al., 2021), SimVLM (Wang et al., 2021)). This approach involves jointly training models on large-scale image and text datasets to learn a shared representation that captures the se- mantic relationship between visual and textual in- puts. Task organizers selected CLIP as a baseline approach. CLIP has emerged as a powerful ap- proach for multimodal understanding and leverages a vast amount of supervision by pre-training on a dataset of 400 million (image, text) pairs collected from …

Content truncated to 500 words