Multimodal Named Entity Recognition for Short Social Media Posts

Proceedings of NAACL-HLT 2018, pages 852–860 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Multimodal Named Entity Recognition for Short Social Media Posts Seungwhan Moon1,2, Leonardo Neves2, Vitor Carvalho3 1 Language Technologies Institute, Carnegie Mellon University 2 Snap Research 3 Intuit seungwhm@cs.cmu.edu, lneves@snap.com, vitor carvalho@intuit.com Abstract We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapchat captions, which comprise short text with ac- companying images. These social media posts often come in inconsistent or incomplete syn- tax and lexical notations with very limited surrounding textual contexts, bringing signif- icant challenges for NER. To this end, we cre- ate a new dataset for MNER called SnapCap- tions (Snapchat image-caption pairs submitted to public and crowd-sourced stories with fully annotated named entities). We then build upon the state-of-the-art Bi-LSTM word/character based NER models with 1) a deep image net- work which incorporates relevant visual con- text to augment textual information, and 2) a generic modality-attention module which learns to attenuate irrelevant modalities while amplifying the most informative ones to ex- tract contexts from, adaptive to each sam- ple and token. The proposed MNER model with modality attention significantly outper- forms the state-of-the-art text-only NER mod- els by successfully leveraging provided visual contexts, opening up potential applications of MNER on myriads of social media platforms. 1 Introduction Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individ- uals. These insights are primarily hidden in un- structured forms of social media posts, such as in free-form text or images without tags. Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a criti- cal step for building structural information, allow- ing for its use in personalized assistance, recom- mendations, advertisement, etc. While many previous approaches (Lample et al., 2016; Ma and Hovy, 2016; Chiu and (a) (b) Figure 1: Multimodal NER + modality attention. (a) Visual contexts help recognizing polysemous entity names (‘Monopoly’ as in a board game versus an eco- nomics term). (b) Modality attention successfully sup- presses word embeddings of a unknown token (‘Marsh- melloooo’ with erroneously trailing ‘o’s), and focuses on character-based context (e.g. capitalized first letter, and lexical similarity to a known named entity (‘Marsh- mello’, a music producer)) for correct prediction. Nichols, 2015; Huang et al., 2015; Lafferty et al., 2001) on NER have shown success for well- formed text in recognizing named entities via word context resolution (e.g. LSTM with word embeddings) combined with character-level fea- tures (e.g. CharLSTM/CNN), several additional challenges remain for recognizing named enti- ties from extremely short and coarse text found in social media posts. For instance, short so- cial media posts often do not provide enough tex- tual contexts to resolve polysemous entities (e.g. “monopoly is da best ”, where ‘monopoly’ may refer to a board game (named entity) or a term in economics). In addition, noisy text includes a huge number of …

Content truncated to 500 words