Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5647–5657, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5647 Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis Dushyant Singh Chauhan, Md Shad Akhtar, Asif Ekbal and Pushpak Bhattacharyya Department of Computer Science & Engineering Indian Institute of Technology Patna Patna, Bihar, India-801106 {1821CS17, shad.pcs15, asif, pb}@iitp.ac.in Abstract In recent times, multi-modal analysis has been an emerging and highly sought-after field at the intersection of natural language process- ing, computer vision, and speech processing. The prime objective of such studies is to lever- age the diversified information, (e.g., textual, acoustic and visual), for learning a model. The effective interaction among these modalities often leads to a better system in terms of per- formance. In this paper, we introduce a re- current neural network based approach for the multi-modal sentiment and emotion analysis. The proposed model learns the inter-modal in- teraction among the participating modalities through an auto-encoder mechanism. We em- ploy a context-aware attention module to ex- ploit the correspondence among the neighbor- ing utterances. We evaluate our proposed ap- proach for five standard multi-modal affect analysis datasets. Experimental results sug- gest the efficacy of the proposed model for both sentiment and emotion analysis over var- ious existing state-of-the-art systems. 1 Introduction In recent past, the world has witnessed tremen- dous growth of various social media platforms, e.g., YouTube, Instagram, Twitter, Facebook, etc. People treat these platforms as a communication medium and freely express themselves with the help of a diverse set of input sources, e.g. videos, images, audio, text etc. The amount of informa- tion produced daily through these mediums are enormous, and hence, the research on multi-modal information processing has attracted attention to the researchers and developers. A video is a multi- modal input which provides visual, acoustic, and textual information. The motivation of multi-modal sentiment and emotion analysis lies in fact to leverage the vari- eties of (often distinct) information from multiple sources for building more efficient systems. For some cases, text can provide a better clue for the prediction, whereas for the others, acoustic or vi- sual sources can be more informative. Similarly, in some situations, a combination of two or more in- formation sources together ensures better and un- ambiguous classification decision. For example, only text ”shut up” can not decide the mood of a person but acoustic (tone of a person) and vi- sual (expression of a person) can reveal the ex- act mood. Similarly, for some instances visual features such as gesture, postures, facial expres- sion etc. have important roles to play in deter- mining the correctness of the system. However, effectively combining this information is a non- trivial task that researchers often have to face (Po- ria et al., 2016; Ranganathan et al., 2016; Lee et al., 2018). Traditionally, ‘text’ has been the key factor in any Natural Language Processing (NLP) tasks, in- cluding sentiment …

Content truncated to 500 words