LT at SemEval-2023 Task 1: Effective Zero-Shot Visual Word Sense Disambiguation Approaches using External Knowledge Sources

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 462–468 July 13-14, 2023 ©2023 Association for Computational Linguistics LT at SemEval-2023 Task 1: Effective Zero-Shot Visual Word Sense Disambiguation Approaches using External Knowledge Sources Florian Schneider and Chris Biemann Language Technology Group, Department of Informatics, Universität Hamburg, Germany {florian.schneider-1, chris.biemann}@uni-hamburg.de Abstract The objective of the SemEval-2023 Task 1: Vi- sual Word Sense Disambiguation (VWSD) (Ra- ganato et al., 2023) is to identify the correct image illustrating the indented meaning of a target word and some minimal additional con- text.The omnipresence of textual and visual data in the task strongly suggests the utiliza- tion of the recent advances in multi-modal ma- chine learning, i.e., pretrained visiolinguistic models (VLMs). Often referred to as founda- tion models due to their strong performance on many vision-language downstream tasks, these models further demonstrate powerful zero-shot capabilities. In this work, we utilize various per- tained VLMs in a zero-shot fashion for multiple approaches using external knowledge sources to enrich the contextual information. Further, we evaluate our methods on the final test data and extensively analyze the suitability of differ- ent knowledge sources, the influence of train- ing data, model sizes, multi-linguality, and dif- ferent textual prompting strategies. Although we are not among the best-performing systems (rank 20 of 56), our experiments described in this work prove competitive results. Moreover, we aim to contribute meaningful insights and propel multi-modal machine learning tasks like VWSD. 1 Introduction This paper presents and analyses effective zero-shot approaches for the SemEval-2023 Task 1: Visual Word Sense Disambiguation (VWSD) (Raganato et al., 2023). In traditional word sense disambigua- tion (WSD), the context or sentence in which am- biguous words, i.e., words with multiple meanings, occur is used for disambiguation by identifying the correct sense in a sense inventory. Frequently used as sense inventories are dictionaries or knowledge bases such as WordNet or DBPedia. As opposed to traditional WSD, in the VWSD shared task, images are used to disambiguate a word given a context. Precisely, given a word and another word serving as context, the task is to identify the image that corresponds to or illustrates the correct meaning in a set of ten images. In the trial phase of the task, 12869 samples in English, including gold la- bels, were provided. However, besides 463 English samples, the final phase test data also contains 305 Italian and 200 Farsi samples. A random VWSD Figure 1: An illustration of a random VWSD sample with the target word ’bonxie’, the context ’bonxie skua’, and the correct image highlighted by a a golden border sample is illustrated in Figure 1. Due to the multi-modal nature of the task, it re- quires methods or models to understand textual semantics contained in the target word and context word and visual semantics contained in the images. Therefore, our approach leverages state-of-the-art pretrained visiolinguistic models (VLMs) in a zero- shot fashion, i.e., we do not continue pretraining or finetune. This is motivated by several reasons based on …

Content truncated to 500 words