RCLN at SemEval-2023 Task 1: Leveraging Stable Diffusion and Image Captions for Visual WSD

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2174–2178 July 13-14, 2023 ©2023 Association for Computational Linguistics RCLN at SemEval-2023 Task 1: Leveraging Stable Diffusion and Image Captions for Visual WSD Antonina Mijatovic and Ekaterina Borisova Ecole Polytechnique Palaiseau, France antonina.mijatovic@polytechnique.edu ekaterina.borisova.2023@polytechnique.edu Davide Buscaldi LIPN, Université Sorbonne Paris Nord Villetaneuse, France buscaldi@lipn.fr Abstract This paper describes the participation of the RCLN team at the Visual Word Sense Disam- biguation task at SemEval 2023. The participa- tion was focused on the use of CLIP as a base model for the matching between text and im- ages with additional information coming from captions generated from images and the gen- eration of images from the prompt text using Stable Diffusion. The results we obtained are not particularly good, but interestingly enough, we were able to improve over the CLIP baseline in Italian by recurring simply to the generated images. 1 Introduction Word Sense Disambiguation (WSD) is a Natural Language Processing task consisting in the identifi- cation of the correct sense of a word having mul- tiple possible meanings, i.e. an ambiguous word, based on the context in which it appears. SemEval- 2023 Task 1 (Raganato et al., 2023) introduces a WSD task that includes a visual representation of the word meaning, named Visual WSD. It consists in determining, out of a set of candidate images, which one is the most pertinent to a given input text. The challenge is that the input text contains an ambiguous word with some of its possible in- terpretations figured in the set of candidate images. For instance, when presented with the context “an- dromeda tree" and the focus word “andromeda", a Visual WSD system should choose an image of the Japanese andromeda plant, rather than an image of the Andromeda galaxy, as the former is a better representation of the meaning of “andromeda" in this context. We approached this task from two directions: the first one was to use image captions to compare to the input text; the second one was to use the input text as a prompt to create an image using a genera- tive model. Image captioning is the task of describ- ing the content of an image in natural language. Various models have been proposed in the past to address this task, from description retrieval to template filling and hand-crafted natural language generation techniques. These techniques have been superseded by modern deep-learning based genera- tive models (Stefanini et al., 2022). Current image captioning models are based on an encoder-decoder architecture. The encoder can be any pre-trained vision transformer, whose purpose is to produce one or multiple one or multiple feature vectors. The decoder is based on a large language model (usually BERT or GPT and derivatives) which uses the embeddings produced in the previous step to produce a sequence of words. In this work, we applied the captioning model that combines the ViT image transformer (Dosovitskiy et al., 2020) with GPT-2 (Radford et al., 2019) as encoder and decoder …

Content truncated to 500 words