OPI PIB at SemEval-2023 Task 1: A CLIP-based Solution Paired with an Additional Word Context Extension

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 482–487 July 13-14, 2023 ©2023 Association for Computational Linguistics OPI PIB at SemEval-2023 Task 1: A CLIP-based Solution Paired with an Additional Word Context Extension Małgorzata Gr˛ebowiec National Information Processing Institute, 00-608 Warsaw, Poland mgrebowiec@opi.org.pl Abstract This article presents our solution for SemEval- 2023 Task 1: Visual Word Sense Disambigua- tion. The aim of the task was to select the most suitable from a list of ten images for a given word, extended by a small textual context. Our solution comprises two parts. The first focuses on an attempt to further extend the textual con- text, based on word definitions contained in WordNet and in Open English WordNet. The second focuses on selecting the most suitable image using the CLIP model with previously developed word context and additional informa- tion obtained from the BEiT image classifica- tion model. Our solution allowed us to achieve a result of 70.84% on the official test dataset for the English language. 1 Introduction Visual word sense disambiguation (VWSD) is a classification problem that involves selecting the one image from a set of candidates that most ac- curately represents the meaning of a target word. This task is important because many words have multiple meanings, and the intended meaning of a word can be ambiguous based on its context. The problem of word ambiguity is a well-established one (Bevilacqua et al., 2021), but VWSD is rela- tively new and contributions to the subject remain sparse. To popularise this issue, SemEval 2023 Task 1: V-WSD: Visual Word Sense Disambigua- tion (Raganato et al., 2023) was organised. The organisers prepared the task for three languages: English, Farsi, and Italian. It was possible to par- ticipate in the competition for each language sep- arately. Our solution is designed exclusively for English. This article describes our solution for the compe- tition. We focus chiefly on describing our attempts to extend the context for given ambiguous words. This is crucial for selecting the most appropriate image. The remainder of the publication is organ- ised as follows: Section 2 describes the datasets provided by the organisers for each phase of the competition; Section 3 presents the design of the system in detail; Section 4 offers information on the evaluation metrics used and describes the ex- periments; Section 5 analyses the results of the experiments; Section 6 contains summarises and presents our conclusions. 2 Dataset description The datasets were prepared by the organisers as TSV files. Each line represents a sample that com- prises a single word, limited textual context, and ten image file names. A separate directory that con- tains image files was attached to each collection. In addition, for the trial and training set, a so-called gold file was included. It contains information about the best-matching image for each word. Af- ter the competition, the organisers released the gold file for the test datasets. Table 1 presents examples of samples from the training and test datasets. The trial …

Content truncated to 500 words