Noam Chomsky at SemEval-2023 Task 4: Hierarchical Similarity-aware Model for Human Value Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 130–135 July 13-14, 2023 ©2023 Association for Computational Linguistics University of Hildesheim at SemEval-2023 Task 1: Combining Pre-trained Multimodal and Generative Models for Image Disambiguation Sebastian Diem, Chan Jong Im, Thomas Mandl University of Hildesheim, Germany mandl@uni-hildesheim.de Abstract Multimodal ambiguity is a challenge for un- derstanding text and images. Large pre-trained models have reached a high level of quality already. This paper presents an implementa- tion for solving an image disambiguation task relying solely on the knowledge captured in multimodal and language models. Within task 1 of SemEval 2023 (Visual Word Sense Disam- biguation), this approach managed to achieve an MRR of 0.738 using CLIP-Large and the OPT model for generating text. Applying a generative model to create more text given a phrase with an ambiguous word leads to an im- provement in our results. The performance gain from a bigger language model is larger than the performance gain from using the larger CLIP model. 1 Introduction Ambiguity is one of the big challenges for Natu- ral Language Processing (NLP) and can often be solved by context. In a multimodal setting, the ambiguity can be solved by identifying the correct visual information. This is the goal of the system implemented in this work. The goal of Task 1: Visual Word Sense Disam- biguation (V-WSD) is to find the most similar image among ten candidate images based on textual input (see Task Overview (Raganato et al., 2023)). The scientific aim is the development of technology for disambiguation in a multimodal context. The textual input consisted of A: the ambigu- ous target word, and B: an additional phrase that clarified the intended meaning of the target word. Some of the candidate images share visual simi- larities, which increases the importance of a precise representation of the model. The task was split into three different languages: English, Farsi, and Ital- ian. The training dataset consists of nearly 13,000 training examples, where each includes an ambigu- ous word, an additional phrase, and ten candidate images. The correct image was located in a sepa- rate file. The test dataset has a similar structure, in- cluding 8,100 images for 463 examples (Raganato et al., 2023). For this work, we focused only on the English dataset. Our approach requires no training and uses only pre-trained models. The progress in large scale language as well as multimodal models in recent years has been rapid. These models capture the semantics of language and images and allow many applications (Gan et al., 2022). The main focus of our approach aims at the re- duction of the ambiguity of the textual input by using "large" generative language models to create more text when given the input phrases. We used four different text inputs, including the phrase, the ambiguous word, the phrase without the ambigu- ous word, and the input created by the generative model. By using the multimodal model CLIP (Rad- ford et al., 2021), we mapped these texts …

Content truncated to 500 words