Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3311–3324, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3311 Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings Yue Wang1, Jing Li2, Michael R. Lyu1, and Irwin King1 1Department of Computer Science and Engineering The Chinese University of Hong Kong, HKSAR, China 2Department of Computing, The Hong Kong Polytechnic University, HKSAR, China 1{yuewang,lyu,king}@cse.cuhk.edu.hk 2jing-amelia.li@polyu.edu.hk Abstract Social media produces large amounts of con- tents every day. To help users quickly capture what they need, keyphrase prediction is receiv- ing a growing attention. Nevertheless, most prior efforts focus on text modeling, largely ignoring the rich features embedded in the matching images. In this work, we explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. To bet- ter align social media style texts and images, we propose: (1) a novel Multi-Modality Multi- Head Attention (M3H-Att) to capture the intri- cate cross-media interactions; (2) image word- ings, in forms of optical characters and image attributes, to bridge the two modalities. More- over, we design a novel unified framework to leverage the outputs of keyphrase classifica- tion and generation and couple their advan- tages. Extensive experiments on a large-scale dataset1 newly collected from Twitter show that our model significantly outperforms the previous state of the art based on traditional co-attentions. Further analyses show that our multi-head attention is able to attend informa- tion from various aspects and boost classifica- tion or generation in diverse scenarios. 1 Introduction The prominent use of social media platforms (such as Twitter) exposes individuals with an abundance of fresh information in a wide variety of forms such as texts, images, videos, etc. Meanwhile, the explosive growth of multimedia data has far out- paced individuals’ capability to understand them, presenting a concrete challenge to digest the mas- sive amount of data, distill the salient contents therein, and provide users with a quick access to the information they need when navigating noisy online data. 1Our code and dataset are released at https://github. com/yuewang-cuhk/CMKP. Post (a): Contemplating the mysteries of life from inside my egg carton...� Post (b): The <mention> have the slight lead at halftime! #cat #cats #CatsOfTwitter #NBAFinals Figure 1: Two multimedia posts from Twitter, where texts offer limited help in identifying their keyphrases while images provide essential clues. To that end, extensive efforts have been made to social media keyphrase prediction2 — aiming to produce a sequence of words that reflect a post’s key concern. Nevertheless, previous work mostly focuses on the use of textual signals (Zhang et al., 2018; Wang et al., 2019a,b), which sometimes pro- vide limited features as social media language is essentially informal and fragmented. To enrich the contexts, here we resort to exploiting the match- ing images, which are widely used in social media posts to deliver auxiliary information from authors (e.g., opinions, feelings, topics, etc.), primarily due to the flourish of mobile Internet. To illustrate our motivation, Figure 1 …

Content truncated to 500 words