SUT at SemEval-2023 Task 1: Prompt Generation for Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2160–2163 July 13-14, 2023 ©2023 Association for Computational Linguistics SUT at SemEval-2023 Task 1: Prompt Generation for Visual Word Sense Disambiguation Omid Ghahroodi⋄∗, Seyed Arshan Dalili⋄∗, Sahel Mesforoush⋄, Ehsaneddin Asgari § ⋄ DH-NLP Lab, Computer Engineering Department, Sharif University of Technology § AI Innovation Center, Data:Lab, Volkswagen AG, Munich, Germany {omid.ghahroodi98, sahel.mesforosh}@sharif.edu, adalili@ce.sharif.edu, asgari@berkeley.edu Abstract Visual Word Sense Disambiguation (V-WSD) identifies the correct visual sense of a multi- sense word in a specific context. This can be challenging as images may need to provide ad- ditional context and words may have multiple senses. A proper V-WSD system can bene- fit applications like image retrieval and cap- tioning. This paper proposes a Prompt Gen- eration approach to solve this challenge. This approach improves the robustness of language- image models like CLIP to contextual ambigu- ities and helps them better correlate between textual and visual contexts of different senses of words. 1 Introduction There are different senses associated with polyse- mous English words (Bevilacqua et al., 2021). The task of determining a specific sense of a word has been one of the most significant challenges in Natu- ral Language Processing. Several studies have been conducted on determining a word’s sense based on its sentence. With the emergence of multimodal works, visual and acoustic information can be used to solve a word’s ambiguity. These models can be used to expand capability of models in limited textual context. Using a proper Visual Word Sense Disambiguation model can contribute to the ad- vancement of different areas, such as image cap- tioning, image retrieval, and other multimodal prob- lem settings. Our main approach is to tackle the problem of V-WSD (Raganato et al., 2023) using prompt en- gineering. We used ChatGPT (OpenAI, 2022) to generate a desired prompt for a given sense. Using CLIP model (Radford et al., 2021) we extract text and image embeddings for the given input and com- pare their cosine distance. Our model works for *The first two authors contributed equally and listed ran- domly. English, Farsi and Italian inputs with average MRR of 71.13% ranking 9th in SemEval final results. Using ChatGPT to expand contextual informa- tion increased model’s accuracy. However, since ChatGPT usage has limitation on number of re- quests, the model input is limited. Our code is publicly available on github1. Related Works There are substantial works on word sense disam- biguation in natural language processing (Navigli, 2009). A word sense disambiguation can be done using supervised (Wang and Wang, 2021), (Bevilac- qua and Navigli, 2019), (Hadiwinoto et al., 2019), or knowledge-based models (Moro et al., 2014), (Agirre et al., 2014). Knowledge-based works use lexicon graph-based information (such as Word- Net) to extract a word’s sense. Supervised methods return the sense of a word based on the presented context (Bevilacqua et al., 2021). The upper bound accuracy for this WSD is 80 percent which is human accuracy on annotated data (Navigli, 2009). State-of-the-art models (Bevilac- qua and …

Content truncated to 500 words