A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6293 - 6302 December 7-11, 2022 ©2022 Association for Computational Linguistics A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity Recognition Baohang Zhou1,2, Ying Zhang1,2,∗, Kehui Song1,2, Wenya Guo1,2, Guoqing Zhao3, Hongbin Wang3, Xiaojie Yuan1,2 1 College of Computer Science, Nankai University, Tianjin, China 2 Tianjin Key Laboratory of Network and Data Security Technology, Tianjin, China 3 Mashang Consumer Finanace Co, Ltd {zhoubaohang,zhangying,songkehui,guowenya}@dbis.nankai.edu.cn {guoqing.zhao02,hongbin.wang02}@msxf.com, yuanxj@nankai.edu.cn Abstract Multimodal named entity recognition (MNER) on social media is a challenging task which aims to extract named entities in free text and incorporate images to classify them into user- defined types. The existing semi-supervised named entity recognition methods focus on the text modal and are utilized to reduce la- beling costs in traditional NER. However, the previous methods are not efficient for semi- supervised MNER. Because the MNER task is defined to combine the text information with image one and needs to consider the mis- match between the posted text and image. To fuse the text and image features for MNER effectively under semi-supervised setting, we propose a novel span-based multimodal varia- tional autoencoder (SMVAE) model for semi- supervised MNER. The proposed method ex- ploits modal-specific VAEs to model text and image latent features, and utilizes product-of- experts to acquire multimodal features. In our approach, the implicit relations between labels and multimodal features are modeled by mul- timodal VAE. Thus, the useful information of unlabeled data can be exploited in our method under semi-supervised setting. Experimen- tal results on two benchmark datasets demon- strate that our approach not only outperforms baselines under supervised setting, but also im- proves MNER performance with less labeled data than existing semi-supervised methods. 1 Introduction Multimodal named entity recognition (MNER) has become a fundamental task to extract named enti- ties from unstructured texts and images on social media (Moon et al., 2018). Compared with tra- ditional named entity recognition (NER), MNER on social media poses the unique challenge that bridging the semantic gap between the posted texts and images is critical to extracting named enti- ties. Therefore, the existing MNER models uti- ∗Corresponding author. Supervised Multimodal Named Entity Recognition Labeled Data Unlabeled Data Semi-supervised Multimodal Named Entity Recognition Figure 1: The settings comparison between supervised and semi-supervised multimodal named entity recogni- tion. For the labeled data, the named entities and their types are highlighted in brackets and different colors. lized cross-modal attention module to fuse the text and image features (Yu et al., 2020; Zhang et al., 2021). Besides, Xu et al. (2022) proposed cross- modal matching and alignment modules to make the representations of the texts and images more consistent. And to retain the useful image infor- mation for MNER, Liu et al. (2022) exploited the two-stage model to refine uncertain labels by fusing the features from the texts and images. To reduce labeling costs in MNER, semi- supervised learning is widely utilized to exploit the useful information of unlabeled data in text modal. …

Content truncated to 500 words