SemEval-2023 Task 1: Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2227–2234 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 1: Visual Word Sense Disambiguation Alessandro Raganato♡ Iacer Calixto♢ Asahi Ushio♣ Jose Camacho-Collados♣ Mohammad Taher Pilehvar♠ ♡ DISCo, University of Milano-Bicocca, Italy ♢ Amsterdam UMC, University of Amsterdam, Netherlands ♣ School of Computer Science and Informatics, Cardiff University, United Kingdom ♠ Tehran Institute for Advanced Studies, Khatam University, Iran ♡alessandro.raganato@unimib.it, ♢i.coimbra@amsterdamumc.nl, ♣ushioa@cardiff.ac.uk, ♣camachocolladosj@cardiff.ac.uk, ♠mp792@cam.ac.uk Abstract This paper presents the Visual Word Sense Dis- ambiguation (Visual-WSD) task. The objective of Visual-WSD is to identify among a set of ten images the one that corresponds to the intended meaning of a given ambiguous word which is accompanied with minimal context. The task provides datasets for three different languages: English, Italian, and Farsi. We received a total of 96 different submissions. Out of these, 40 systems outperformed a strong zero-shot CLIP- based baseline (Radford et al., 2021). Partic- ipating systems proposed different zero- and few-shot approaches, often involving genera- tive models and data augmentation. More in- formation can be found on the task’s website: https://raganato.github.io/vwsd/. 1 Introduction Word Sense Disambiguation (WSD) is the task of associating a word in context with its intended sense, generally from a pre-defined sense inven- tory. While there has been significant progress in the last few years (Bevilacqua et al., 2021; Loureiro et al., 2021), mainly powered by progress in lan- guage models, WSD has been mainly limited to settings with textual content only. In most settings, WSD uses sense inventories obtained from lexical resources, such as WordNet (Miller, 1998). How- ever, in real-world scenarios, WSD is often asso- ciated with other modalities, such as images. The Visual Word Sense Disambiguation (Visual-WSD) task aims at filling this gap: given a word and some limited textual context (often a single word), the task is to select among a set of candidate images the one which corresponds to the intended meaning of the target word. Figure 1 provides a simplified overview of the task. To make the task more challenging, the dataset is constructed with the following objectives in mind: Figure 1: Three examples for the Visual-WSD task. The target (ambiguous) word (in bold) is provided with minimal context (one or two words). The task is to associate the intended sense with the relevant image (underlined). Note: For the task, nine negative images were provided, but for the sake of simplicity, we only show three negative images. (1) at least one image associated with an incorrect meaning of the word is listed among the options; (2) minimal context (one or two words) is provided for the target word to increase the disambiguation diffi- culty; and (3) other negative examples are sampled from the same domain as the target sense. Objec- tive (1) guarantees that the task pinpoints systems’ abilities in not only identifying the correct class, but also in doing that at the more fine-grained word sense level. Objective (2) ensures a challenging disambiguation setting in which …

Content truncated to 500 words