P-MNER: Cross Modal Correction Fusion Network with Prompt Learning for Multimodal Named Entity Recognitiong

P-MNER: Cross Modal Correction Fusion Network with Prompt Learning for Multimodal Named Entity Recognitiong Zhuang Wang1, Yijia Zhang1*, Kang An1, Xiaoying Zhou1, Mingyu Lu2*, Hongfei Lin3 1College of Information Science and Technology, Dalian Maritime University / Dalian, China 2College of Artificial Intelligence, Dalian Maritime University / Dalian, China 3College of Computer Science and Technology, Dalian University of Technology / Dalian, China {wang_1120211498,zhangyijia,1120221416_ankang,zhouxiaoying}@dlmu.edu.cn lumingyu@dlmu.edu.cn hflin@dlut.edu.cn Abstract Multimodal Named Entity Recognition (MNER) is a challenging task in social media due to the combination of text and image features. Previous MNER work has focused on predicting entity information after fusing visual and text features. However, pre-training language models have already acquired vast amounts of knowledge during their pre- training process. To leverage this knowledge, we propose a prompt network for MNER tasks (P-MNER).To minimize the noise generated by irrelevant areas in the image, we design a visual feature extraction model (FRR) based on FasterRCNN and ResNet, which uses fine-grained visual features to assist MNER tasks. Moreover, we introduce a text correction fusion module (TCFM) into the model to address visual bias during modal fusion. We employ the idea of a residual network to modify the fused features using the original text features.Our experiments on two benchmark datasets demonstrate that our proposed model outperforms existing MNER methods. P-MNER’s ability to leverage pre-training knowledge from language models, incorporate fine-grained visual features, and correct for visual bias, makes it a promising approach for multimodal named entity recognition in social media posts. 1 Introduction With the rapid development of the Internet, social media platforms have experienced an expo- nential growth of content. These platforms offer a wealth of user-generated posts that provide valuable insights into the events, opinions, and preferences of both individuals and groups. Named Entity Recognition (NER) is a crucial task in which entities contained in textual data are detected and mapped to predefined entity types, such as location (LOC), person (PER), organization (ORG), and miscellaneous (MISC). Incorporating visual information from posts has been shown to significantly enhance the accuracy of entity prediction from social media content. For instance, as illustrated in Fig.1, the sentence ”Alban got Rikard a snowball in the snow” can be easily resolved by leveraging the visual cues in the accompanying image, allowing us to identify ”Rikard” as an animal. However, relying solely on textual data to predict entities may lead to erroneous predictions, such as identifying ”Rikard” as a name. With the continuous evolution of deep learning models, several multi-modal Named En- tity Recognition (NER) models have been proposed to enhance the prediction performance of entities by incorporating visual information. These models employ techniques such as cross- attention(Wu et al., 2020; Zhang et al., 2018), adversarial learning(Goodfellow et al., 2014; Frankle and Carbin, 2018), and graph fusion(Xiao et al., 2021; Wu et al., 2020). However, previous methods fused text features with visual features and directly fed them into a neural network model for prediction. This approach overlooks the wealth of information embedded This work is supported by the National Natural …

Content truncated to 500 words