GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1592–1597 July 13-14, 2023 ©2023 Association for Computational Linguistics GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images Shibingfeng Zhang1,2 and Shantanu Nath1 and Davide Mazzaccara1 1CIMeC, University of Trento 2Saarland University {shibingfeng.zhang,shantanu.nath,davide.mazzaccara}@unitn.it Abstract Given a word in context, the task of Visual Word Sense Disambiguation consists of select- ing the correct image among a set of candidates. To select the correct image, we propose a so- lution blending text augmentation and multi- modal models. Text augmentation leverages the fine-grained semantic annotation from Word- Net to get a better representation of the tex- tual component. We then compare this sense- augmented text to the set of image using pre- trained multimodal models CLIP and ViLT. Our system has been ranked 16th for the English language, achieving 68.5 points for hit rate and 79.2 for mean reciprocal rank. The code to this project is available on Github1. 1 Introduction Many and very common words in language are pol- ysemous, i.e., they have more than one meanings associated with the lexical object (Ježek, 2015). The task of Word Sense Disambiguation (WSD) consists of identify the correct meaning of a word in a specific context, its sense. For example, the word Andromeda has three different meanings ac- cording to the Cambridge dictionary: the princess from the Greek mythology, the large galaxy and a particular plant with flowers. The objective of WSD is to determine which sense is active in a specific context, such as the sentence “We found the Andromeda growing on the banks of a lagoon of fresh water”. Visual Word Sense Disambiguation (VWSD) is the extension of the traditional WSD task that in- corporates visual information in addition to textual context (Gella et al., 2016). In SemEval-2023 Task 1 proposed by Raganato et al. 2023, the objective is to identify the correct image from a pool of can- didates that corresponds to the intended meaning 1https://github.com/Zhangshibf/ SemEval2023Task1_Visual_Word_Sense_ Disambiguation Figure 1: Different images for sense of the words An- dromeda. The correct image is the one on the left. of the word in context. For example, the word is Andromeda, the context is Andromeda tree, and three candidate images are presented in Figure 1. The image located on the left displays a photograph of an Andromeda tree, which is the correct choice, whereas the others images show other objects and are therefore incorrect. We tackle this task with an elaborated text aug- mentation strategy and the power of pre-trained language-vision models. In particular, our text aug- mentation strategy employs the fine-grained lin- guistic resource of WordNet (Miller, 1994) to get better textual representations. Multimodal models are then leveraged to select the correct image in the pool of candidates in a zero-shot. Our system has been ranked 16th for the English language (50 sub- missions, 125 participants), achieving 68.5 points for hit rate and 79.2 for mean reciprocal rank. We also achieved 37.7 for hit rate and 55.7 for …

Content truncated to 500 words