StFX NLP at SemEval-2023 Task 1: Multimodal Encoding-based Methods for Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 409–414 July 13-14, 2023 ©2023 Association for Computational Linguistics StFX NLP at SemEval-2023 Task 1: Multimodal Encoding-based Methods for Visual Word Sense Disambiguation Yuchen Wei Department of Computer Science St. Francis Xavier University x2020fct@stfx.ca Milton King Department of Computer Science St. Francis Xavier University mking@stfx.ca Abstract SemEval-2023’s Task 1, Visual Word Sense Dis- ambiguation, a task about text semantics and visual semantics, is about selecting the best-matched im- age to represent a target word in a limited context. We explored several methods, including image cap- tioning methods and CLIP-based methods, and sub- mitted our predictions in the competition for this task. This paper will focus on the methods we used and their performance, and provide an analysis and discussion of their performance. 1 Introduction SemEval-2023’s Task 1: Visual Word Sense Dis- ambiguation (V-WSD) involves selecting an image from a list of candidates, that best exhibits a given target word in a small context. In this task (Ra- ganato et al., 2023), each sample will contain one target word, a limited context, and ten candidate images. The ten candidate images contain one gold image - the image that best matches the sense of the target word in its context. In addition to this, the ten candidate images also contain images related to other senses (i.e., the meaning of the word) of the target word as well as images not related to the target word. The limited context contains two or three words (including the target word), and the vast majority of these contexts are strongly related to a sense of the target word. Figure 1 shows one example in the dataset. In this example, the target word is andromeda and the context is andromeda tree. The first image is the gold image (the tree with white flowers). The second image (whale) and the third image (snake) are some of the candidate images not related to the target word. The fourth image (stars) is the candidate image related to an- other sense of the word andromeda. The dataset for this task contains a trial set, a training set, and a testing set. The context and target words in the samples of the trial set and the training set are in En- glish. The testing set contains samples in English, Italian, and Farsi. Figure 1: The sample andromeda. . This task involves both natural language pro- cessing and computer vision. This is because we need to acquire semantic features of both text and images. For this task, we chose to map text and images to the same vector space and then compare them using a similarity metric. To perform this feature mapping, we require the use of models that have been trained on other datasets. We approached this task with several pre-trained visual language models. This includes the image caption model (Wang et al., 2022) and Contrastive Language- Image Pre-training (CLIP) (Radford et al., 2021) models. 2 Related Work Word sense …

Content truncated to 500 words