Rutgers Multimedia Image Processing Lab at SemEval-2023 Task-1: Text-Augmentation-based Approach for Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1483–1490 July 13-14, 2023 ©2023 Association for Computational Linguistics Rutgers Multimedia Image Processing Lab at SemEval-2023 Task-1: Text-Augmentation-based Approach for Visual Word Sense Disambiguation Keyi Li1, Sen Yang2, Chenyang Gao1, Ivan Marsic1 1Rutgers University 2Waymo LLC {kl734,sy358,cg694,marsic}@rutgers.edu Abstract This paper describes our system used in SemEval-2023 Task-1: Visual Word Sense Dis- ambiguation (VWSD). The VWSD task is to identify the correct image that corresponds to an ambiguous target word given limited tex- tual context. To reduce word ambiguity and en- hance image selection, we proposed several text augmentation techniques, such as prompting, WordNet synonyms, and text generation. We experimented with different vision-language pre-trained models to capture the joint features of the augmented text and image. Our approach achieved the best performance using a combi- nation of GPT-3 text generation and the CLIP model. On the multilingual test sets, our sys- tem achieved an average hit rate (at top-1) of 51.11 and a mean reciprocal rank of 65.69. 1 Introduction Polysemous words are common in human language. These words are ambiguous and can be interpreted in variant ways under different contexts. Although it is easy for humans to distinguish different word senses, machines need to transform the word senses into data structure and analyze the differences. Word sense disambiguation (WSD) is a task for ma- chines to identify the meaning of words given lim- ited text contexts (Navigli, 2009; Bevilacqua et al., 2021). WSD is widely used in different NLP tasks, including information retrieval, machine transla- tion, information extraction, content analysis, and lexicography. With the rapid growth of the multi- modal datasets, WSD task has expanded from lan- guage to the visual field, which aims to improve the tasks such as image description, visual question answering, object detection, and image retrieval (Chen et al., 2015; Gella et al., 2016; de Guevara et al., 2020; Calabrese et al., 2021). For exam- ple, when we search for an image of “Andromeda tree” with an ambiguous word “Andromeda”, the search engine needs to identify the meaning of “An- dromeda” under the context “tree”, i.e., a species of plant instead of the Greek goddess or the galaxy. The SemEval-2023 Task-1 describes this task as the visual word sense disambiguation task (VWSD): given an ambiguous target word with limited tex- tual context, select among a set of candidate images the one which corresponds to the intended meaning of the target word (Raganato et al., 2023). Word embedding has been widely used to rep- resent words in numerical vectors for machines to interpret the word sense (Mikolov et al., 2013). A word is transformed into high-dimensional vector space where each dimension represents a unique feature of the word. Word embeddings can capture the semantic and syntactic relations between words, e.g., words with similar meanings will be closer in the space, and antonyms words will be orthogonal to each other. However, word embeddings have a limitation that polysemous words with multiple meanings cannot be captured by …

Content truncated to 500 words