PMCoders at SemEval-2023 Task 1: RAltCLIP: Use Relative AltCLIP Features to Rank

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1751–1755 July 13-14, 2023 ©2023 Association for Computational Linguistics PMCoders at SemEval-2023 Task 1: RAltCLIP: Use Relative AltCLIP Features to Rank Mohammad Javad Pirhadi, Motahhare Mirzaei, Mohammad Reza Mohammadi and Sauleh Eetemadi Iran University of Science and Technology at Tehran, Iran {mohammad_pirhadi, m_mirzaei96}@comp.iust.ac.ir, {mrmohammadi, sauleh}@iust.ac.ir Abstract Visual Word Sense Disambiguation (VWSD) task aims to find the most related image among 10 images to an ambiguous word in some lim- ited textual context. In this work, we use Alt- CLIP features and a 3-layer standard trans- former encoder to compare the cosine sim- ilarity between the given phrase and differ- ent images. Also, we improve our model’s generalization by using a subset of LAION- 5B. The best official baseline achieves 37.20% and 54.39% macro-averaged hit rate and MRR (Mean Reciprocal Rank) respectively. Our best configuration reaches 39.61% and 56.78% macro-averaged hit rate and MRR respectively. The code will be made publicly available on GitHub. 1 Introduction WSD (Word Sense Disambiguation) is the task of identifying which sense of a word is meant in a sentence or other segment of text. In the VWSD (visual-WSD) task (Raganato et al., 2023), given a potentially ambiguous word and some limited textual context, the image corresponding to the intended meaning should be selected from a set of candidate images. Such a model can operate better as a text-to-image retrieval system. CLIP (Radford et al., 2021) uses a contrastive loss function to map text and image features into a common space by minimizing the distance between the related caption and image and maximizing the distance between negative pairs. AltCLIP (Chen et al., 2022) alters the language encoder in CLIP and uses XL-R LM (Language Model) to extend its language capabilities. We use AltCLIP as the base model as the dataset includes Italian and Farsi languages in addition to English. The official dataset consists of 12869, 16, 463, 305, and 200 samples as training, trial, English test, Italian test, and Farsi test data respectively. Each sample has a word (potentially ambiguous), a phrase including the word, and 10 images. To improve model generalization, we also use a subset of the LAION-5B (Schuhmann et al., 2022) dataset which consists of 5 billion image-caption pairs and has 3 parts: LAION-2B-en, LAION-2B-multi, and LAION-1B-nolang. The experiments results show that the proposed model which relatively compares the images can be used to improve text-to-image retrieval systems by re-ranking the most related images. 2 System Description Figure 1 shows an overview of the proposed model and the following subsections discuss it in more detail. 2.1 Feature Extraction We map the phrase and images into a common space using AltCLIP (the AltCLIP model is frozen). In this stage, we have 11 vectors, 1 for the phrase and 10, each corresponding to an image. 2.2 Word Sense Disambiguation "You shall know a word by the company it keeps" - J. R. Firth (1957) Getting inspiration from this famous quote, we provide the …

Content truncated to 500 words