ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3176 - 3189 July 10-15, 2022 ©2022 Association for Computational Linguistics ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition Xinyu Wang♢‡, Min Gui♡, Yong Jiang♠∗, Zixia Jia♢, Nguyen Bach♣, Tao Wang, Zhongqiang Huang♠, Fei Huang♠, Kewei Tu♢∗ ♢School of Information Science and Technology, ShanghaiTech University ♢Shanghai Engineering Research Center of Intelligent Vision and Imaging ♠DAMO Academy, Alibaba Group ♡Shopee, Singapore ♣Microsoft {wangxy1,jiazx,tukw}@shanghaitech.edu.cn, min.gui@shopee.com {yongjiang.jy,z.huang,f.huang}@alibaba-inc.com nguyenbach@microsoft.com Abstract Recently, Multi-modal Named Entity Recog- nition (MNER) has attracted a lot of attention. Most of the work utilizes image information through region-level visual representations ob- tained from a pretrained object detector and relies on an attention mechanism to model the interactions between image and text represen- tations. However, it is difficult to model such interactions as image and text representations are trained separately on the data of their re- spective modality and are not aligned in the same space. As text representations take the most important role in MNER, in this paper, we propose Image-text Alignments (ITA) to align image features into the textual space, so that the attention mechanism in transformer- based pretrained textual embeddings can be better utilized. ITA first aligns the image into regional object tags, image-level captions and optical characters as visual contexts, concate- nates them with the input texts as a new cross- modal input, and then feeds it into a pretrained textual embedding model. This makes it easier for the attention module of a pretrained textual embedding model to model the interaction be- tween the two modalities since they are both represented in the textual space. ITA further aligns the output distributions predicted from the cross-modal input and textual input views so that the MNER model can be more practi- cal in dealing with text-only inputs and robust to noises from images. In our experiments, we show that ITA models can achieve state-of- the-art accuracy on multi-modal Named Entity Recognition datasets, even without image in- formation.1 ∗Yong Jiang and Kewei Tu are the corresponding authors. ‡: This work was done when Xinyu Wang, Min Gui and Nguyen Bach were at Alibaba Group. 1Our code is publicly available at https://github. com/Alibaba-NLP/KB-NER/ITA. 1 Introduction Named Entity Recognition (NER) (Sundheim, 1995) has attracted increasing attention in natu- ral language processing community. It has been applied to a lot of domains such as news (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003), E-commerce (Fetahu et al., 2021), social me- dia (Strauss et al., 2016; Derczynski et al., 2017) and bio-medicine (Do˘gan et al., 2014; Li et al., 2016). Several recent studies focus on improving the accuracy of NER models through utilizing im- age information (MNER) in tweets (Zhang et al., 2018; Moon et al., 2018; Lu et al., 2018). Most ap- proaches to MNER use the attention mechanism to model the interaction between image and text repre- sentations (Yu et al., 2020; Zhang et al., 2021a; Sun et al., …

Content truncated to 500 words