SRCB at SemEval-2023 Task 1: Prompt Based and Cross-Modal Retrieval Enhanced Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 439–446 July 13-14, 2023 ©2023 Association for Computational Linguistics SRCB at SemEval-2023 Task 1: Prompt Based and Cross-Modal Retrieval Enhanced Visual Word Sense Disambiguation Xudong Zhang�, Tiange Zhen∗, Jing Zhang, Yujin Wang, Song Liu Samsung Research China - Beijing (SRC-B) {xudong.z1, tiange.zhen, jing97.zhang, yujin.wang, s0101.liu}@samsung.com Abstract The Visual Word Sense Disambiguation (VWSD) shared task aims at selecting the im- age among candidates that best interprets the semantics of a target word with a short-length phrase for English, Italian, and Farsi. The lim- ited phrase context, which only contains 2-3 words, challenges the model’s understanding ability, and the visual label requires image-text matching performance across different modali- ties. In this paper, we propose a prompt based and multimodal retrieval enhanced VWSD sys- tem, which uses the rich potential knowledge of large-scale pretrained models by prompting and additional text-image information from knowl- edge bases and open datasets. Under the En- glish situation and given an input phrase, (1) the context retrieval module predicts the cor- rect definition from sense inventory by match- ing phrase and context through a biencoder architecture. (2) The image retrieval module retrieves the relevant images from an image dataset. (3) The matching module decides that either text or image is used to pair with im- age labels by a rule-based strategy, then ranks the candidate images according to the similar- ity score. Our system ranks first in the English track and second in the average of all languages (English, Italian, and Farsi). 1 Introduction Word Sense Disambiguation (WSD) is a challeng- ing task of Natural Language Processing, which aims at identifying the polysemic word with cor- rect interpretation (Bevilacqua et al., 2021). It has numerous applications, such as machine transla- tion, search engines, and text understanding. The traditional WSD task provides the predicting word with its context, which is supposed to be a sentence containing the word, and chooses the proper sense from the predefined inventories (Miller, 1995; Nav- igli and Ponzetto, 2012). The model is asked to ∗Contribution during Internship in Samsung Research China-Beijing. extract the semantic feature of the target word from the particular text and match it with the most prac- tical sense. The WSD task can be processed as a classification problem among the senses as la- bels in the given inventory by taking advantage of the neural language model’s powerful semantic un- derstanding capability. The appearance of Visual WSD connects the different modalities and gives the word sense a new presentation form. The sense inventory now is a set of images instead of text def- initions, and the target word has to be paired with the corresponding image. The Vision-Language model, CLIP (Radford et al., 2021), has shown its outstanding performance on unifying image-test modalities, and it provides potential knowledge for semantic understanding required by disambigua- tion. The SemEval-2023 Task-1 Visual Word Sense Disambiguation (VWSD) (Raganato et al., 2023) is a multilingual WSD task, for English, Italian, and Farsi, with the …

Content truncated to 500 words