Dual-Encoder Transformers with Cross-modal Alignment for Multimodal Aspect-based Sentiment Analysis

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 414–423 November 20–23, 2022. ©2022 Association for Computational Linguistics 414 Dual-Encoder Transformers with Cross-modal Alignment for Multimodal Aspect-based Sentiment Analysis Zhewen Yu†, Jin Wang†∗, Liang-Chih Yu‡∗ and Xuejie Zhang† †School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China ‡Department of Information Management, Yuan Ze University, Taiwan Contact:wangjin@ynu.edu.cn, lcyu@saturn.yzu.edu.tw Abstract Multimodal aspect-based sentiment analysis (MABSA) aims to extract the aspect terms from text and image pairs, and then analyze their corresponding sentiment. Recent studies typi- cally use either a pipeline method or a unified transformer based on a cross-attention mecha- nism. However, these methods fail to explicitly and effectively incorporate the alignment be- tween text and image. Supervised finetuning of the universal transformers for MABSA still requires a certain number of aligned image-text pairs. This study proposes a dual-encoder trans- former with cross-modal alignment (DTCA). Two auxiliary tasks, including text-only extrac- tion and text-patch alignment are introduced to enhance cross-attention performance. To align text and image, we propose an unsuper- vised approach which minimizes the Wasser- stein distance between both modalities, forc- ing both encoders to produce more appropriate representations for the final extraction. Experi- mental results on two benchmarks demonstrate that DTCA consistently outperforms existing methods. For reproducibility, the code for this paper is available at: https://github. com/windforfurture/DTCA. 1 Introduction Human experience of the world is multimodal, e.g., seeing objects, hearing sounds, feeling textures, and tasting flavors. Multimodal experiences are usually mutually associated to some extent. For ex- ample, images are usually associated with tags and text explanations, and text often contains images to more clearly express the main intent of the author. With the widespread availability of smart phones with digital cameras, social media posts have be- come increasingly multimodal . To practically ap- ply the existing aspect-based sentiment analysis, one must be able to interpret such multimodal at- tributes together (Yu et al., 2022; Ling et al., 2022). ∗Corresponding authors. [ Kevin Durant ] says [ Kyrie Irving ] has more skill than [Allen Iverson]. What do health heroes look like ? [ Dr Lucille Corti ] died [ AIDS ] 1996, [ Dr Lukwiya ] died [Ebola] 2000. (a) (Kevin Durant, Neutral) (Kyrie Irving, Positive) (Allen Iverson, Negative) (Dr Lucille Corti, Positive) (AIDS, Negative) (Dr Lukwiya, Positive) (Ebola, Negative) (b) Figure 1: Two examples of joint multimodal aspect sentiment analysis. Figure 1 (a) shows an example: What do health heroes look like? Dr Lucille Corti died AIDS 1996, Dr Lukwiya died Ebola 2000. An intelligent sys- tem is expected to extract four aspect-sentiment pairs from this text, i.e., (Dr Lucille Corti, posi- tive), (AIDS, negative), (Dr Lukwiya, positive) and (Ebola, negative). Notably, if only the language modality is used for inference, the model tends to predict (Dr Lucille Corti, negative) and (Dr Luk- wiya, negative). Related to the vision modality, the expression of the text …

Content truncated to 500 words