Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 417–422 July 13-14, 2023 ©2023 Association for Computational Linguistics Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval Xiaomeng Pan, Zhousi Chen, Mamoru Komachi∗ Tokyo Metropolitan University {pan-xiaomeng@ed., chen-zhousi@ed., komachi@}tmu.ac.jp Abstract In this study, we propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation (WSD) component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). We confirmed previous findings — human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts yield the worst results. Our code is available at https:// github.com/pxm427/WSD-for-IR. 1 Introduction The SemEval 2023 shared task VWSD1 combines WSD and Image Retrieval (IR), which aims to se- lect a correct image among ten candidates using a phrase containing ambiguous words. Neural mod- els are likely to be attracted by frequent tokens, labels, and senses of ambiguous words, particularly in limited contexts. We determined that CLIP (Rad- ford et al., 2021) fails to find the correct images us- ing phrases with the ambiguous words of frequent senses even enhanced with contrastive learning on large-scale data. As shown in Figure 1, given phrase “Andromeda tree”, the pretrained CLIP selected incorrect im- ages that focused on either constellation “An- dromeda” or part of “tree”, neglecting the phrase’s meaning entirely. This sample demonstrates am- biguity as a challenge for state-of-the-art neural models. Therefore, we exploited GPT-3 (Brown et al., 2020), a large language model (LLM), for its pretrained knowledge as implicit sense disam- biguation and phrase context enrichment for this task. Prompt engineering boosts model performance and plays a crucial role in applying LLMs to many NLP tasks, which lessens training and testing dis- crepancies resulting from human-like languages ∗Now at Hitotsubashi University 1https://raganato.github.io/vwsd/ 3.Andromeda tree 1.Andromeda 7.Ebenaceae 4.Erythrostemon gilliesii 2.Euphorbiaceae 5.Gentiana triflora 8.ambystoma mexicanum 9.Orcinus orca 6. Helianthus 10.Crotalus Figure 1: CLIP ranks ten images with respective rele- vance to the phrase “Andromeda tree”, which consists of ambiguous word “Andromeda” and limited context “tree”. The goal is to select the correct image from ten images of different relevance corresponding to the in- tended meaning of “Andromeda tree”. Note that the first (blue) and second (green) images ranked higher than the correct third image (red). (Liu et al., 2023). A prompt refers to a text or a set of instructions that guides the model to generate a specific type of response or output. A prompt can be a question, statement, keyword, or sequence of words that provide context and information to the model. Recent research on prompt techniques demonstrates that well-designed prompts spur the potential of neural models without modifying their parameters (Jin et al., 2022). We are curious about how prompts can improve the performance of CLIP on VWSD. Our main contributions are as follows: 1. We explored different templates for queries and observed their effects on VWSD. The quotes and highlighting …

Content truncated to 500 words