TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 506–511 July 13-14, 2023 ©2023 Association for Computational Linguistics TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation Qihao Yang†, Yong Li†, Xuelin Wang‡, Shunhao Li†, Tianyong Hao†∗ †School of Computer Science, South China Normal University, Guangzhou, China ‡College of Chinese Language and Culture, Jinan University, Guangzhou, China {charlesyeung, lycutter, haoty}@m.scnu.edu.cn wangxuelin@stu2022.jnu.edu.cn, lishunhao99@foxmail.com Abstract Visual Word Sense Disambiguation (WSD), as a fine-grained image-text retrieval task, aims to identify the images that are relevant to am- biguous target words or phrases. However, the difficulties of limited contextual informa- tion and cross-linguistic background knowl- edge in text processing make this task chal- lenging. To alleviate this issue, we propose a Fine-grained Contrastive Language-Image Learning (FCLL) model, which learns fine- grained image-text knowledge by employing a new fine-grained contrastive learning mech- anism and enriches contextual information by establishing relationship between concepts and sentences. In addition, a new multimodal- multilingual knowledge base involving am- biguous target words is constructed for visual WSD. Experiment results on the benchmark datasets from SemEval-2023 Task 1 show that our FCLL ranks at the first in overall evaluation with an average H@1 of 72.56% and an average MRR of 82.22%. The results demonstrate that FCLL is effective in inference on fine-grained language-vision knowledge. Source codes and the knowledge base are publicly available at https://github.com/CharlesYang030/FCLL. 1 Introduction The problem of word polysemy, first recognized in machine translation, is one of challenging tasks in Natural Language Processing (NLP). In the field of NLP, resolving word polysemy is generally re- garded as Word Sense Disambiguation (WSD), and it still remains as one of the most challenging and pervasive linguistic phenomena in NLP at present (Bevilacqua et al., 2021). To promote the research of the issues, international competitions have been held such as SemEval1 to advance the state-of-the- art in word sense analysis and to help create high- quality annotated datasets in WSD. Since SemEval- *Corresponding author. 1https://semeval.github.io/ 2023, visual WSD has been introduced and it con- tains three tracks including English, Farsi and Ital- ian (Raganato et al., 2023). Given a target word and some limited textual context, the visual WSD task is to select the one which relevant to the intended meaning of the word among a set of candidate im- ages. Theoretically, the general WSD task aims at making explicit the semantics of a word in context by identifying the most suitable meaning (called sense) from a pre-defined sense inventory (Bevilac- qua et al., 2021), which only involves words and senses. Differently, visual WSD requires models to learn more fine-grained image-text knowledge to establish the relationship among words, senses and images. Meanwhile, the limitations of visual WSD are significant. Because it is difficult in most cases to determine the specific sense of a word from a phrase containing few words without additional contextual information, and even more difficult to further identify the images related to the word. …

Content truncated to 500 words