UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2043–2051 July 13-14, 2023 ©2023 Association for Computational Linguistics UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation Michael Ogezi, Bradley Hauer, Talgat Omarov, Ning Shi, Grzegorz Kondrak Alberta Machine Intelligence Institute Department of Computing Science University of Alberta, Edmonton, Canada {mikeogezi,bmhauer,omarov,ning.shi,gkondrak}@ualberta.ca Abstract We describe the systems of the University of Al- berta team for the SemEval-2023 Visual Word Sense Disambiguation (V-WSD) Task. We present a novel algorithm that leverages glosses retrieved from BabelNet, in combination with text and image encoders. Furthermore, we com- pare language-specific encoders against the ap- plication of English encoders to translated texts. As the contexts given in the task datasets are extremely short, we also experiment with aug- menting these contexts with descriptions gener- ated by a language model. This yields substan- tial improvements in accuracy. We describe and evaluate additional V-WSD methods which use image generation and text-conditioned image segmentation. Overall, the results of our offi- cial submission rank us 18 out of 56 teams. Some of our unofficial results are even bet- ter than the official ones. Our code is pub- licly available at https://github.com/ UAlberta-NLP/v-wsd. 1 Introduction This paper addresses our work on SemEval-2023 Task 1: Visual Word Sense Disambiguation1 (Ra- ganato et al., 2023). The V-WSD task is closely related to WSD, and similarly involves understand- ing and classifying the meaning of a polysemous word in context. The distinction is in how classes are defined: In WSD, a system has access to a sense inventory that enumerates the possible senses of each word, and the task is to classify the focus word according to the sense that best corresponds to its intended meaning. In V-WSD, a system is given a set of candidate images, and the task is to select the image which depicts the intended meaning of the focus word. The multi-modal nature of V-WSD introduces challenges not encountered in WSD. First, image 1https://raganato.github.io/vwsd/ Figure 1: The task is to select the image that best rep- resents the meaning of the focus word (e.g., bat) in the context (e.g., “baseball bat.”) processing is generally more computationally inten- sive than text processing. Second, a V-WSD system must represent the meanings of both images and text, and must have mechanisms to compare these multi-modal semantic representations. Last, since the candidate images in V-WSD are not restricted to a sense inventory, they may exhibit highly vari- able levels of sense granularity. The V-WSD task is motivated by cases where textual context alone is insufficient to disambiguate a word. In such cases, visual context may be avail- able to facilitate disambiguation. For example, the word play is ambiguous in the context “that was a good play,” as it may refer to a theatrical per- formance or an action in a sport. However, an associated image of a stage or a sports field would enable a V-WSD system to disambiguate play. We propose a novel V-WSD algorithm …

Content truncated to 500 words