Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2149 - 2159 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis Yan Ling, Jianfei Yu∗, and Rui Xia∗ School of Computer Science and Engineering, Nanjing University of Science and Technology, China {ylin, jfyu, rxia}@njust.edu.cn Abstract As an important task in sentiment analysis, Multimodal Aspect-Based Sentiment Analysis (MABSA) has attracted increasing attention in recent years. However, previous approaches either (i) use separately pre-trained visual and textual models, which ignore the cross- modal alignment or (ii) use vision-language models pre-trained with general pre-training tasks, which are inadequate to identify fine- grained aspects, opinions, and their alignments across modalities. To tackle these limitations, we propose a task-specific Vision-Language Pre-training framework for MABSA (VLP- MABSA), which is a unified multimodal encoder-decoder architecture for all the pre- training and downstream tasks. We further de- sign three types of task-specific pre-training tasks from the language, vision, and multi- modal modalities, respectively. Experimen- tal results show that our approach generally outperforms the state-of-the-art approaches on three MABSA subtasks. Further analysis demonstrates the effectiveness of each pre- training task. The source code is publicly re- leased at https://github.com/NUSTM/ VLP-MABSA. 1 Introduction Recent years have witnessed increasing attention on the Multimodal Aspect-Based Sentiment Anal- ysis (MABSA) task1. Previous research mostly focused on its two subtasks, including Multimodal Aspect Term Extraction (MATE) and Multimodal Aspect-oriented Sentiment Classification (MASC). Given a text-image pair as input, MATE aims to extract all the aspect terms mentioned in the text (Zhang et al., 2018; Lu et al., 2018; Wu et al., 2020a,b; Zhang et al., 2021a), whereas MASC ∗Corresponding authors. 1The MABSA task is also known as Target-Oriented Mul- timodal Sentiment Analysis or Entity-Based Multimodal Sen- timent Analysis in the literature. Image Text Sergio Ramos chosen as the best player of UCL final Output (Sergio Ramos, Positive) (UCL, Neutral) Table 1: An example of the MABSA task aims to classify the sentiment towards each ex- tracted aspect term (Xu et al., 2019; Yu and Jiang, 2019; Khan and Fu, 2021). As the two sub- tasks are closely related to each other, Ju et al. (2021) recently introduced the Joint Multimodal Aspect-Sentiment Analysis (JMASA) task, aiming to jointly extract the aspect terms and their corre- sponding sentiments. For example, given the text- image pair in Table. 1, the goal of JMASA is to identify all the aspect-sentiment pairs, i.e., (Sergio Ramos, Positive) and (UCL, Neutral). Most of the aforementioned studies to MABSA primarily focused on employing pre-trained uni- modal models (e.g., BERT for text and ResNet for image) to obtain textual and visual features respectively. The separate pre-training of visual and textual features ignores the alignment between text and image. It is therefore crucial to perform vision-language pre-training to capture such cross- modal alignment. However, for the MABSA task, the studies on vision-language pre-training are still lacking. To the best of our knowledge, there are very few studies focusing on vision-language …

Content truncated to 500 words