teamPN at SemEval-2023 Task 1: Visual Word Sense Disambiguation Using Zero-Shot MultiModal Approach

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 457–461 July 13-14, 2023 ©2023 Association for Computational Linguistics teamPN at SemEval-2023 Task 1: Visual Word Sense Disambiguation Using Zero-Shot MultiModal Approach Nikita Katyal ∗ Siam Makro Pawan Rajpoot ∗ MUST Research Subhanandh Tamilarasu ∗ Rakuten India Pvt. Ltd. Joy Mustafi MUST Research Abstract Visual Word Sense Disambiguation shared task at SemEval-2023 aims to identify an image cor- responding to the intended meaning of a given ambiguous word (with related context) from a set of candidate images. The lack of textual de- scription for the candidate image and the corre- sponding word’s ambiguity makes it a challeng- ing problem. This paper describes teamPN’s multi-modal and modular approach to solving this in English track of the task. We efficiently used recent multi-modal pre-trained models backed by real-time multi-modal knowledge graphs to augment textual knowledge for the images and select the best matching image ac- cordingly. We outperformed the baseline model by 5 points and proposed a unique approach that can further work as a framework for other modular and knowledge-backed solutions. 1 Introduction As per SemEval-2023 task 1 definition for Visual- WSD Raganato et al. (2023). "Given a word and some limited textual context, the task is to select among a set of candidate images the one which corresponds to the intended meaning of the target word." The final evaluation data in this task has 3 language tracks English, Italian, and Farsi. An initial trial data set was released with 16 sam- ples, whereas the final training dataset was released with 12869 samples, where each pair of text and contextual text were mapped to a target image that has to be retrieved from a set of 10 images. The images attached to each of the training samples overlap with other samples. And thus, a total of 12999 unique images were found in the training data pool. These images vary from a wide vari- ety of genres. A sample input data example can be seen in fig. 1. Here, ’white’ is the term, and ’white yolk’ is the context. The model has to re- trieve the right image from the set of input images. *Equal Contribution The text consists of words and context covering Figure 1: Example visual word sense disambiguation, the ambiguous word in this example is "white" and the context is "white yolk". Here the positive image is high- lighted by green and negative images are highlighted in pink. various domains; even the context’s intentions vary with different samples. A word cloud of the text involved in the train data can be seen in 2. How- ever, a significant proportion of the dataset looks to retrieve images based on their scientific terms, especially plants, birds and rocks. This can be clearly interpreted from the word cloud containing the terms genus, family, herb, shrub, and minerals. Figure 2: Word cloud of dataset 2 Related Work Starting from Radford et al. (2021) there has been a recent increase in multi-modal …

Content truncated to 500 words