Stronger Baseline for Robust Results in Multimodal Sentiment Analysis

Stronger Baseline for Robust Results in Multimodal Sentiment Analysis Wei Yang National Institute of Advanced Industrial Science and Technology (AIST), Japan wei.yang@aist.go.jp Jun Ogata National Institute of Advanced Industrial Science and Technology (AIST), Japan jun.ogata@aist.go.jp Abstract Significant progress in sentiment analysis and emotion recognition has been made recently, and there has also been an increase in the re- quirements for solving real-world problems. However, human language is complicated and multimodal, making it difficult for computers or artificial intelligence systems to understand. In this study, we adopt several self-supervised learning models to strengthen the learning of representations for multi-modalities (i.e., lan- guage, acoustic and visual modalities) to im- prove the performance of sentiment analysis systems. We also recommend effective fu- sion and ensemble methods that are easy to apply to obtain a stronger performance. In this study, considering the preliminary exper- iments, we use a freely available sentiment analysis benchmark, CMU-MOSI. Our rec- ommended techniques and constructed mul- timodal sentiment analysis systems empiri- cally demonstrate the improvements in the experiment results by approximately 9% and 10.4% for binary classification and multi- classification, respectively, in terms of accu- racy, and a 32% reduction in the mean abso- lute error. 1 Introduction Sentiment analysis (also known as opinion mining) is a task that aims to systematically identify, ex- tract, quantify, and study emotional states. Sen- timent analysis and emotion recognition are both key applications of artificial intelligence (AI). AI has also become indispensable to tasks involving human-computer interaction (HCI). Some related studies have demonstrated their state-of-the-art (SOTA) result evaluations based on or compared with the sentiment analysis systems constructed using low-level features for single or multiple modalities. Further, it is challenging to determine the amount of improvement that can be achieved using a newly proposed method in com- parison to more robust systems built using high- level representations and how effective the method is. This indicates the necessity to know whether the proposed technique or the artificial neural net- works (ANNs) can solve new problems or slightly improve the results that have already been addressed or achieved elsewhere. The main focus of this study was to construct a robust and reliable multimodal sentiment analy- sis system using several artificial techniques from three perspectives: (1) high-level representations for multi-modalities; (2) robust neural network architec- ture with an effective fusion strategy; (3) easily ap- plicable and available for solving real-world prob- lems in different fields, such as customer service and clinical medicine (Valstar et al., 2016; Venek et al., 2017). This study introduces three specific techniques for strengthening the performance of multimodal sen- timent analysis and empirically demonstrates how they are applied to improve the experimental results. We also investigate how these techniques improve the performance of sentiment analysis, either singly used or fusion with each other. We also discuss how these empirical experiments and findings can be used to understand the factors affecting the sys- tem construction. Our recommended techniques in- clude: • Utilize self-supervised learning (SSL) models as feature extractors for every single …

Content truncated to 500 words