Grounded Multimodal Named Entity Recognition on Social Media

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9141–9154 July 9-14, 2023 ©2023 Association for Computational Linguistics Grounded Multimodal Named Entity Recognition on Social Media Jianfei Yu∗, Ziyan Li∗, Jieming Wang and Rui Xia† School of Computer Science and Engineering, Nanjing University of Science and Technology, China {jfyu, zyanli, wjm, rxia}@njust.edu.cn Abstract In recent years, Multimodal Named Entity Recognition (MNER) on social media has at- tracted considerable attention. However, ex- isting MNER studies only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction and insufficient for entity disambiguation. To solve these is- sues, in this work, we introduce a Grounded Multimodal Named Entity Recognition (GM- NER) task. Given a text-image social post, GMNER aims to identify the named entities in text, their entity types, and their bounding box groundings in image (i.e., visual regions). To tackle the GMNER task, we construct a Twitter dataset based on two existing MNER datasets. Moreover, we extend four well-known MNER methods to establish a number of base- line systems and further propose a Hierarchical Index generation framework named H-Index, which generates the entity-type-region triples in a hierarchical manner with a sequence-to- sequence model. Experiment results on our annotated dataset demonstrate the superiority of our H-Index framework over baseline sys- tems on the GMNER task. Our dataset anno- tation and source code are publicly released at https://github.com/NUSTM/GMNER. 1 Introduction Fueled by the rise of phones and tablets with cam- era functions, user posts on social media platforms such as Twitter are increasingly multimodal, e.g., containing images in addition to text. The explo- sive growth of multimodal posts is far beyond hu- mans’ capability to digest them. Hence, it presents a pressing need for automatically extracting im- portant information such as entities and relations from the large amount of multimodal posts, which is crucial for structured knowledge graph construc- tion to help people efficiently understand massive ∗Equal contribution. † Corresponding author. • Michael Jordan, PER • the Fields Institute, ORG Michael Jordan is now in Toronto, giving his distinguished lecture at the Fields Institute. Text Image • Toronto, LOC • Michael Jordan, PER, • the Fields Institute, ORG, Text Image Michael Jordan is now in Toronto, giving his distinguished lecture at the Fields Institute. • Toronto, LOC, None (a) Mulitmodal NER (b) Grounded Mulitmodal NER Figure 1: Comparison between Multimodal NER (MNER) and our Grounded Multimodal NER (GMNER) task. GMNER is a task to identify the named entities in text, their entity types, and their bounding box groundings in image. None denotes there is no grounded bounding box for the entity Toronto. content. As an emerging subtask for multimodal knowledge graph construction (Liu et al., 2019), Multimodal Named Entity Recognition (MNER) on social media has recently attracted increasing attention (Zhang et al., 2018; Moon et al., 2018). Given a text-image social post, MNER aims to rec- ognize named entities in text and classify them into pre-defined types such as person (PER), location (LOC), …

Content truncated to 500 words