Rahul Patil at SemEval-2023 Task 1: V-WSD: Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1271–1275 July 13-14, 2023 ©2023 Association for Computational Linguistics Rahul Patil at SemEval-2023 Task 1: V-WSD: Visual Word Sense Disambiguation Rahul Patil†, Pinal Patel∗, Charin Patel∗, Mangal Verma∗ ∗RAAPID.ai †HEALTHCARENLP SOFTECH LLP †{rahul.p}@healthcarenlp.com, ∗{pinal,charin,mangal}@raapid.ai Abstract Semeval 2023 task 1: Visual-WSD, In this pa- per, we propose an ensemble of two Neural network systems that ranks 10 images given a word and limited textual context. We have used OpenAI CLIP based models for the English language and multilingual text-to-text transla- tion models for Farsi-to-English and Italian-to- English. Additionally, we propose a system that learns from multilingual bert-base embed- dings for text and ResNet101 embeddings for the image. Considering all three languages into account this system has achieved the fourth rank. 1 Introduction Visual word sense disambiguation (Visual-WSD) (Raganato et al., 2023) is an important task for understanding and measuring machine’s ability to find correct image from a set of candidate images given the word and the limited text context. Word sense disambiguation (WSD) has long been formulated and studied as an important prob- lem in natural language processing since 1940s. Visual Word Sense Disambiguation (Visual-WSD) is a combination of natural language processing and computer vision that involves identifying the correct visual image of an ambiguous word using limited context. For example, the word "bank" can refer to a financial institution or the side of a river. In the context of limited text data, such as “bank erosion”, Visual-WSD aims to identify the image displaying the correct meaning of the word in its context. Visual-WSD is an important task in natu- ral language processing and computer vision, as it helps to improve the accuracy of systems that use text and visual information together, such as im- age captioning, video analysis and visual question answering. Successful Visual-WSD can help im- prove the accuracy of natural language processing applications that rely on visual information. We used an ensemble of two different ap- proaches 1) Fine-tune CLIP (Radford et al., 2021) base architecture 2) Joint Embedding. The first ap- proach involves finetuning CLIP based model with additional data and the second approach involves mapping image and text-based features into a com- mon vector space, which allows direct comparison of the two features. This enables the model to learn the relationship between visual and text features and use this knowledge to determine the correct sense of a word. The proposed system has achieved the 4th overall rank in this competition with the hit rate of 69.805 and mean reciprocal rank score of 78.230. The system has achieved the second rank in English language with the HIT rate of 83.153 and MRR rate of 88.802, first rank in Italian with the HIT rate of 84.262 and MRR rate of 89.048, lastly achieved 16th rank in Farsi language with the HIT rate of 42.000 and MRR rate of 56.841. The system lacked understanding of Farsi language and struggle with Farsi to English conversion. 2 Background Three …

Content truncated to 500 words