ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1463–1469 July 13-14, 2023 ©2023 Association for Computational Linguistics ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation Clifton A. Poth∗ Martin B. Hentschel∗ Tobias Werner Hannah Sterz Leonard Bongard Technical University of Darmstadt {cliftonalexander.poth, martin.hentschel}@stud.tu-darmstadt.de Abstract Successful word sense disambiguation (WSD) is a fundamental element of natural language understanding. As part of SemEval-2023 Task 1, we investigate WSD in a multimodal setting, where ambiguous words are to be matched with candidate images representing word senses. We compare multiple systems based on pre-trained CLIP models. In our experiments, we find CLIP to have solid zero-shot performance on monolingual and multilingual data. By em- ploying different fine-tuning techniques, we are able to further enhance performance. However, transferring knowledge between data distribu- tions proves to be more challenging. 1 Introduction Determining the correct meaning of an ambiguous word in a given textual context is a fundamental element of successful understanding of natural lan- guage. Formulated in the task of Word Sense Dis- ambiguation (WSD), it remains a key challenge of Natural Language Processing (NLP). Recent successful approaches tackling this task often lever- age large pre-trained language models (Bevilacqua et al., 2021). With the rise of large pre-trained vision-and-language (V&L) models such as CLIP (Radford et al., 2021), the question arises how well these models can tackle WSD in a multimodal set- ting. Towards answering this question, Task 1 of SemEval-2023 proposes a visual WSD task (Ra- ganato et al., 2023), combining ambiguous con- textualized words with images encoding different word senses. Instead of selecting the correct word sense from a sense inventory, a system now has to retrieve the best matching image from a set of can- didate images, thereby bridging the gap between vision and language. In this paper, we present our contributions to the SemEval-2023 Visual-WSD task. Our work ∗* equal contribution focuses on empirically comparing the WSD perfor- mance of multiple CLIP models in different trans- fer learning settings. This includes the investigation of zero-shot transfer and adapters (Rebuffi et al., 2017; Houlsby et al., 2019) as a parameter-efficient means of transfer learning. We especially aim to highlight to which degree simple techniques are suf- ficient for successful task performance. Our best submitted system ranks 9th out of 54 on the official task leaderboard (counting only one submission per person). 2 Background 2.1 Word Sense Disambiguation (WSD) WSD is the task of determining the exact meaning of an ambiguous word in a given context. Typi- cally, this is achieved by selecting the most suitable word sense from a pre-defined, static word sense inventory such as WordNet (Miller et al., 1990). Existing solutions for this task mainly follow two approaches: Knowledge-based approaches exploit the knowledge encoded into lexical resources, such as the graph structure of WordNet. Supervised ap- proaches aim to learn a parameterized model, such as a neural network, that directly predicts the cor- rect sense s given a word w and a context …

Content truncated to 500 words