RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER

Proceedings of the 28th International Conference on Computational Linguistics, pages 1852–1862 Barcelona, Spain (Online), December 8-13, 2020 1852 RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER Lin Sun1, Jiquan Wang2, Yindu Su2, Fangsheng Weng1, Yuxuan Sun1, Zengwei Zheng1, and Yuanyi Chen1 1 Department of Computer Science, Zhejiang University City College, Hangzhou, China {sunl,zhengzw,chenyuanyi@zucc.edu.cn}@zucc.edu.cn 2 College of Computer Science and Technology, Zhejiang University, Hangzhou, China {wangjiquan,yindusu}@zju.edu.cn Abstract Multimodal named entity recognition (MNER) for tweets has received increasing attention re- cently. Most of the multimodal methods used attention mechanisms to capture the text-related visual information. However, unrelated or weakly related text-image pairs account for a large proportion in tweets. Visual clues unrelated to the text would incur uncertain or even negative effects for multimodal model learning. In this paper, we propose a novel pre-trained multimodal model based on Relationship Inference and Visual Attention (RIVA) for tweets. The RIVA model controls the attention-based visual clues with a gate regarding the role of image to the seman- tics of text. We use a teacher-student semi-supervised paradigm to leverage a large unlabeled multimodal tweet corpus with a labeled data set for text-image relation classification. In the multimodal NER task, the experimental results show the significance of text-related visual fea- tures for the visual-linguistic model and our approach achieves SOTA performance on the MNER datasets. 1 Introduction Social media such as Twitter has become a part of many people’s everyday lives. It is an important source for various applications such as open event extraction (Wang et al., 2019), social knowledge graph (Hos- seini, 2019). Named entity recognition (NER) for tweets is the first key task of these applications. NER has achieved excellent performance on news articles; however, the recognition results in tweets are still not satisfactory (Akbik et al., 2018; Akbik et al., 2019). One of the reasons is that tweets are short mes- sages and the context for inference is insufficient. Recent works on tweets based on multimodal learning have been increasing (Moon et al., 2018; Lu et al., 2018; Zhang et al., 2018; Arshad et al., 2019). The researchers attempted to improve the performance of NER in tweets with the aid of visual clues. Most of the multimodal NER (MNER) methods used attention weights to extract visual clues related to the NEs (Lu et al., 2018; Zhang et al., 2018; Arshad et al., 2019) . The visual attention-based models always assume that the images in tweets are related to the texts, such as words in the text are represented in the image, e.g., Figure 1(a) shows a successful visual attention example from Lu et al. (2018). In fact, texts and images in tweets have diverse relations. Vempala and Preot¸iuc-Pietro (2019) categorized text-image relationship according to whether “Image adds to the tweet meaning”. “Image adds to the tweet meaning” represents the role of image to the semantics of text in tweets. The type of “Image does not add to the tweet meaning” account for approximately 56% in the Vempala’s Bloomberg …

Content truncated to 500 words