Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal Relation Detection

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4395–4405 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4395 Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal Relation Detection Xincheng Ju1, Dong Zhang1∗, Rong Xiao2, Junhui Li1, Shoushan Li1, Min Zhang 1, Guodong Zhou1 1 School of Computer Science and Technology, Soochow University, China 2 Alibaba Group, China xcju@stu.suda.edu.cn, xiaorong.xr@taobao.com, {dzhang, jhli, lishoushan, minzhang, gdzhou}@suda.edu.cn Abstract Aspect terms extraction (ATE) and aspect sen- timent classification (ASC) are two fundamen- tal and fine-grained sub-tasks in aspect-level sentiment analysis (ALSA). In the textual anal- ysis, jointly extracting both aspect terms and sentiment polarities has been drawn much at- tention due to the better applications than in- dividual sub-task. However, in the multi- modal scenario, the existing studies are limited to handle each sub-task independently, which fails to model the innate connection between the above two objectives and ignores the better applications. Therefore, in this paper, we are the first to jointly perform multi-modal ATE (MATE) and multi-modal ASC (MASC), and we propose a multi-modal joint learning ap- proach with auxiliary cross-modal relation de- tection for multi-modal aspect-level sentiment analysis (MALSA). Specifically, we first build an auxiliary text-image relation detection mod- ule to control the proper exploitation of visual information. Second, we adopt the hierarchi- cal framework to bridge the multi-modal con- nection between MATE and MASC, as well as separately visual guiding for each sub mod- ule. Finally, we can obtain all aspect-level sentiment polarities dependent on the jointly extracted specific aspects. Extensive experi- ments show the effectiveness of our approach against the joint textual approaches, pipeline and collapsed multi-modal approaches. 1 Introduction Multi-modal aspect-level (aka target-oriented) sen- timent analysis (MALSA) is an important and fine- grained task in multi-modal sentiment analysis (MSA). Previous studies normally cast MALSA in social media as two independent sub-tasks: Multi- modal Aspect Terms Extraction (MATE) and Multi- modal Aspect Sentiment Classification (MASC). First, MATE aims to detect a set of all potential ∗ Corresponding Author [NBA]Neu: [Spurs]Pos rout [Thunder]Neg | Tempo Sports RT @ funnytwittingg : [OBAMA]Neg TO [ISRAEL]Neu ? [OBAMA]Neg TO [UKRAINE] ? [OBAMA]Neg TO [USA]Neu ? (a) (b) Figure 1: Two examples for joint multi-modal aspect- sentiment analysis. aspect terms from a free text with its accompany- ing image (Wu et al., 2020a). Second, MASC aims to classify the sentiment polarity of a multi-modal post towards a given aspect in textual modality (Yu and Jiang, 2019). To better satisfy the practical applications, the aspect term-polarity co-extraction, which solves ATE and ASC simultaneously, receives much atten- tion recently in a textual scenario (Wan et al., 2020; Chen and Qian, 2020b; Ying et al., 2020). How- ever, to our best knowledge, in the multi-modal scenario, the joint MATE and MASC, i.e., joint multi-modal aspect-sentiment analysis (JMASA), have never been investigated so far. For this joint multi-modal task, we believe that there exist the following challenges at least. On the one hand, visual modality may provide no clues for one of sub-tasks. For example, …

Content truncated to 500 words