Face-Sensitive Image-to-Emotional-Text Cross-modal Translation for Multimodal Aspect-based Sentiment Analysis

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3324 - 3335 December 7-11, 2022 ©2022 Association for Computational Linguistics Face-Sensitive Image-to-Emotional-Text Cross-modal Translation for Multimodal Aspect-based Sentiment Analysis Hao Yang, Yanyan Zhao∗and Bing Qin Harbin Institute of Technology hyang@ir.hit.edu.cn, yyzhao@ir.hit.edu.cn, qinb@ir.hit.edu.cn Abstract Aspect-level multimodal sentiment analysis, which aims to identify the sentiment of the target aspect from multimodal data, recently has attracted extensive attention in the commu- nity of multimedia and natural language pro- cessing. Despite the recent success in textual aspect-based sentiment analysis, existing mod- els mainly focused on utilizing the object-level semantic information in the image but ignore explicitly using the visual emotional cues, espe- cially the facial emotions. How to distill visual emotional cues and align them with the textual content remains a key challenge to solve the problem. In this work, we introduce a face- sensitive image-to-emotional-text translation (FITE) method, which focuses on capturing vi- sual sentiment cues through facial expressions and selectively matching and fusing with the target aspect in textual modality. To the best of our knowledge, we are the first that explicitly utilize the emotional information from images in the multimodal aspect-based sentiment anal- ysis task. Experiment results show that our method achieves state-of-the-art results on the Twitter-2015 and Twitter-2017 datasets. The improvement demonstrates the superiority of our model in capturing aspect-level sentiment in multimodal data with facial expressions1. 1 Introduction As an important task of multimodal sentiment analysis, multimodal aspect-based sentiment analy- sis(MABSA) aims to classify the sentiment polar- ity expressed in the sentence-image pair towards a specific aspect. In contrast to textual aspect-level sentiment analysis, mining and utilizing emotional clues for the aspect in visual content is the core problem. Early works (Xu et al., 2019; Yu and Jiang, 2019; Wang et al., 2021) on the MABSA task treat the ∗* Corresponding author 1Our code is publicly available at: https://github.com/ yhit98/FITE. Figure 1: Examples of multimodal aspect-based senti- ment analysis with image caption. visual features extracted by unimodal pre-trained models as equal to the textual features. These works focus on utilizing the attention mechanism to implicitly align and fuse the semantic information and the emotional information in the two modal- ities. More recently, Yu et al. (2019) takes the object-level visual semantic information into ac- count in the feature-based multimodal fusion stage. As shown in Figure 1(a) and (b), we observe that the facial expression in the image is important to identify the sentiment of the target aspect “antonel- laRoccuzzo” and “Kate Middleton”. But due to the limitations of the dataset and weak supervi- sion of cross-modal alignment in the MABSA task, the methods that implicitly capture emotional in- formation in images are more likely to learn the non-emotional cues bias. Khan and Fu (2021) introduce an object-aware transformer to translate images to captions for cross-modal fusion. The cross-modal translation effectively solves the problem of multimodal fea- ture fusion from different feature spaces. As shown in Figure 1, the introduced image caption repre- sents semantic information …

Content truncated to 500 words