PoliTo at SemEval-2023 Task 1: CLIP-based Visual-Word Sense Disambiguation Based on Back-Translation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1447–1453 July 13-14, 2023 ©2023 Association for Computational Linguistics PoliTo at SemEval-2023 Task 1: CLIP-based Visual-Word Sense Disambiguation Based on Back-Translation Lorenzo Vaiani and Luca Cagliero and Paolo Garza Politecnico di Torino {lorenzo.vaiani,luca.cagliero,paolo.garza}@polito.it Abstract Visual-Word Sense Disambiguation (V-WSD) entails resolving the linguistic ambiguity in a text by selecting a clarifying image from a set of (potentially misleading) candidates. In this paper, we address V-WSD using a state- of-the-art Image-Text Retrieval system, namely CLIP. We propose to alleviate the linguistic ambiguity across multiple domains and lan- guages via text and image augmentation. To augment the textual content we rely on back- translation with the aid of a variety of auxiliary languages. The approach based on fine-tuning CLIP on the full phrases is effective in accu- rately disambiguating words and incorporating back-translation enhances the system’s robust- ness and performance on the test samples writ- ten in Indo-European languages. 1 Introduction Visual-Word Sense Disambiguation (V-WSD) is a seminal task in the field of multimodal learn- ing involving both natural language processing and computer vision (Raganato et al., 2023). It entails the identification of the correct sense of a given text containing ambiguous words by selecting a clarifying image among a set of candidates, pos- sibly including misleading options. Tackling the aforesaid task is relevant to address well-known real-world challenges such as multimodal machine translation (Barrault et al., 2018; Gella et al., 2019), image caption generation (Wang et al., 2020a; Pra- manick et al., 2022), visual search and object recog- nition (Yao et al., 2019; Zou et al., 2023). The key issues in V-WSD are the linguistic am- biguity of the expressions occurring in the text, which may subtend multiple senses, and the high variability of the candidate images across differ- ent domains, which makes the problem of image retrieval non-trivial. To address the above issues, we propose a novel approach to V-WSD based on CLIP (Radford et al., 2021), i.e., a state-of-the- art vision-language model trained on massive data consisting of image-caption pairs. Leveraging the outstanding capability of CLIP to capture associa- tions between visual and textual content, we first fine-tune the pretrained model on the raw V-WSD data to assess model robustness across multiple do- mains and languages. Next, we explore the use of data augmentation on both text and images, where text is augmented via back-translation. Consid- ering text translations in multiple languages has the twofold aim of enriching the model with new text variants generated via machine translation and making the CLIP model suitable for multilingual scenarios. We also empirically analyze the effect of considering different snippets of the input text, i.e., the ambiguous words only, the main topic de- scription, or the full phrase. The results confirm the suitability of the fine- tuned CLIP version to effectively tackle the V- WSD by extracting the key information from the full phrases. Specifically, they show a performance boost on the family of Indo-European languages that are predominant in the …

Content truncated to 500 words