Tensor Fusion Network for Multimodal Sentiment Analysis

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Tensor Fusion Network for Multimodal Sentiment Analysis Amir Zadeh†, Minghai Chen† Language Technologies Institute Carnegie Mellon University {abagherz,minghail}@cs.cmu.edu Soujanya Poria Temasek Laboratories, NTU, Singapore sporia@ntu.edu.sg Erik Cambria School of Computer Science and Engineering, NTU, Singapore cambria@ntu.edu.sg Louis-Philippe Morency Language Technologies Institute Carnegie Mellon University morency@cs.cmu.edu Abstract Multimodal sentiment analysis is an in- creasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a mul- timodal setup where other relevant modal- ities accompany language. In this paper, we pose the problem of multimodal senti- ment analysis as modeling intra-modality and inter-modality dynamics. We intro- duce a novel model, termed Tensor Fusion Network, which learns both such dynam- ics end-to-end. The proposed approach is tailored for the volatile nature of spoken language in online videos as well as ac- companying gestures and voice. In the ex- periments, our model outperforms state-of- the-art approaches for both multimodal and unimodal sentiment analysis. 1 Introduction Multimodal sentiment analysis (Morency et al., 2011; Zadeh et al., 2016b; Poria et al., 2015) is an increasingly popular area of affective comput- ing research (Poria et al., 2017) that focuses on generalizing text-based sentiment analysis to opin- ionated videos, where three communicative modal- ities are present: language (spoken words), visual (gestures), and acoustic (voice). This generalization is particularly vital to part of the NLP community dealing with opinion min- ing and sentiment analysis (Cambria et al., 2017) since there is a growing trend of sharing opinions in videos instead of text, specially in social media (Facebook, YouTube, etc.). The central challenge in multimodal sentiment analysis is to model the inter-modality dynamics: the interactions between † means equal contribution Figure 1: Unimodal, bimodal and trimodal interac- tion in multimodal sentiment analysis. language, visual and acoustic behaviors that change the perception of the expressed sentiment. Figure 1 illustrates these complex inter-modality dynamics. The utterance “This movie is sick” can be ambiguous (either positive or negative) by itself, but if the speaker is also smiling at the same time, then it will be perceived as positive. On the other hand, the same utterance with a frown would be per- ceived negatively. A person speaking loudly “This movie is sick” would still be ambiguous. These examples are illustrating bimodal interactions. Ex- amples of trimodal interactions are shown in Fig- ure 1 when loud voice increases the sentiment to strongly positive. The complexity of inter-modality dynamics is shown in the second trimodal exam- ple where the utterance “This movie is fair” is still weakly positive, given the strong influence of the word “fair”. A second challenge in multimodal sentiment analysis is efficiently exploring intra-modality dy- namics of a specific modality (unimodal interac- tion). Intra-modality dynamics are particularly 1103 challenging for the language analysis since mul- timodal sentiment analysis is performed on spo- ken language. A spoken opinion such as “I think it was alright ...Hmmm ...let …

Content truncated to 500 words