Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3342–3352 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3342 Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer Jianfei Yu1, Jing Jiang2, Li Yang3, and Rui Xia1,∗ 1 School of Artificial Intelligence, Nanjing University of Science & Technology, China 2 School of Information Systems, Singapore Management University, Singapore 3 DBS Bank, Singapore {jfyu, rxia}@njust.edu.cn, jingjiang@smu.edu.sg, liyang@dbs.com Abstract In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts. Existing approaches for MNER mainly suffer from two drawbacks: (1) despite gener- ating word-aware visual representations, their word representations are insensitive to the vi- sual context; (2) most of them ignore the bias brought by the visual context. To tackle the first issue, we propose a multimodal interac- tion module to obtain both image-aware word representations and word-aware visual repre- sentations. To alleviate the visual bias, we fur- ther propose to leverage purely text-based en- tity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions. Experiments show that our uni- fied approach achieves the new state-of-the-art performance on two benchmark datasets. 1 Introduction Recent years have witnessed the explosive growth of user-generated contents on social media plat- forms such as Twitter. While empowering users with rich information, the flourish of social media also solicits the emerging need of automatically ex- tracting important information from these massive unstructured contents. As a crucial component of many information extraction tasks, named entity recognition (NER) aims to discover named enti- ties in free text and classify them into pre-defined types, such as person (PER), location (LOC) and organization (ORG). Given its importance, NER has attracted much attention in the research com- munity (Yadav and Bethard, 2018). Although many methods coupled with either dis- crete shallow features (Zhou and Su, 2002; Finkel et al., 2005; Torisawa et al., 2007) or continuous deep features (Lample et al., 2016; Ma and Hovy, ∗Corresponding author. (a). [Kevin Durant PER] enters [Oracle Arena LOC] wearing off — White x [Jordan MISC] (b). Vote for [King of the Jungle MISC] — [Kian PER] or [David PER] ? Figure 1: Two examples for Multimodal Named Entity Recognition (MNER). Named entities and their entity types are highlighted. 2016) have shown success in identifying entities in formal newswire text, most of them perform poorly on informal social media text (e.g., tweets) due to its short length and noisiness. To adapt existing NER models to social media, various methods have been proposed to incorporate many tweet-specific features (Ritter et al., 2011; Li et al., 2012, 2014; Limsopatham and Collier, 2016). More recently, as social media posts become increasingly multi- modal, several studies proposed to exploit useful visual information to improve the performance of NER (Moon et al., 2018; Zhang et al., 2018; Lu et al., 2018). In this work, following the recent trend, we focus on multimodal named entity recognition …

Content truncated to 500 words