SzegedAI at SemEval-2023 Task 1: Applying Quasi-Symbolic Representations in Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1965–1971 July 13-14, 2023 ©2023 Association for Computational Linguistics SzegedAI at SemEval-2023 Task 1: Applying Quasi-Symbolic Representations in Visual Word Sense Disambiguation Gábor Berend♠♢ ♠ELKH-SZTE Research Group on Artificial Intelligence ♢University of Szeged, Hungary berendg@inf.u-szeged.hu Abstract In this paper, we introduce our submission in the task of visual word sense disambiguation (V-WSD). Our proposed solution operates by deriving quasi-symbolic semantic categories from the hidden representations of multi-modal text-image encoders. Our results are mixed, as we manage to achieve a substantial boost in performance when evaluating on a valida- tion set, however, we experienced detrimen- tal effects during evaluation on the actual test set. Our positive results on the validation set confirms the validity of the quasi-symbolic fea- tures, whereas our results on the test set re- vealed that the proposed technique was not able to cope with the sufficiently different distribu- tion of the test data. 1 Introduction Multi-modal text-image encoders, such as CLIP (Radford et al., 2021) have a demonstrated capa- bility of performing zero-shot image classification tasks over diverse benchmark datasets. The con- trastive learning objective applied during the pre- training of such models make them a natural choice for the V-WSD shared task (Raganato et al., 2023), where the goal is to choose from a set of candidate images the one which suits the best some query expression including a potentially ambiguous term. That is, the shared task problem can be framed as an image retrieval task, for where the transfer learn- ing capabilities of CLIP can be arguably exploited. The kind of continuous hidden representations produced by neural models, including CLIP, are limited by their lack of translucency from a hu- man perspective. Our earlier work demonstrated that by using an unsupervised technique relying on sparse coding, it becomes possible to connect static word embedding representations with human interpretable concepts and properties (Balogh et al., 2020). In (Berend et al., 2018), we successfully em- ployed these representations for hypernym discov- ery (Camacho-Collados et al., 2018), surpassing the performance obtained by relying on traditional and less interpretable dense word embeddings. In (Berend, 2020), we demonstrated that the technique can be extended to contextualized word represen- tations, substantially improving the performance in classic unimodal, text-only word sense disam- biguation. Our approach for inducing implicit semantic in- formation to words within their context built on the observation that performing sparse coding over the contextual representations of pre-trained lan- guage models result in representations that align well with semantic categories. With our shared task participation, our goal was to investigate if similar benefits that we saw for the classic uni-modal, text- only WSD task can be observed for multi-modal CLIP-based representations in V-WSD. Our experiments delivered mixed results, as the sparse multi-modal representations provided sub- stantial performance gain during the development phase, when evaluation was carried out on a held- out part of the training set, but the same quasi- symbolic features proved to be detrimental for the …

Content truncated to 500 words