Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

Proceedings of NAACL-HLT 2019, pages 370–379 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 370 Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis Md Shad Akhtar†, Dushyant Singh Chauhan†, Deepanway Ghosal†, Soujanya Poria+, Asif Ekbal† and Pushpak Bhattacharyya† † Department of Computer Science & Engineering Indian Institute of Technology Patna, India {shad.pcs15, 1821CS17, deepanway.ee14, asif, pb}@iitp.ac.in + School of Computer Science and Engineering, Nanyang Technological University, Singapore sporia@ntu.edu.sg Abstract Related tasks often have inter-dependence on each other and perform better when solved in a joint framework. In this paper, we present a deep multi-task learning framework that jointly performs sentiment and emotion analysis both. The multi-modal inputs (i.e., text, acoustic and visual frames) of a video convey diverse and distinctive information, and usually do not have equal contribution in the decision making. We propose a context- level inter-modal attention framework for si- multaneously predicting the sentiment and ex- pressed emotions of an utterance. We evalu- ate our proposed approach on CMU-MOSEI dataset for multi-modal sentiment and emotion analysis. Evaluation results suggest that multi- task learning framework offers improvement over the single-task framework. The proposed approach reports new state-of-the-art perfor- mance for both sentiment analysis and emo- tion analysis. 1 Introduction With the rapid growth of social media video plat- forms such as Youtube, Vimeo, users now tend to upload videos on these platforms. Such video platforms offer users an opportunity to express their opinions on any topic. Videos usually con- sist of audio and visual modalities, and thus can be considered as a source of multi-modal infor- mation. Although videos contain more informa- tion than text, fusing multiple modalities is a ma- jor challenge. A common practice in sentiment analysis and emotion recognition or affective com- puting, in general, is to analyze textual opinions. However, in recent days multi-modal affect anal- ysis has gained a major attention (Poria et al., 2017b, 2016). In these works, in addition to the visual frames, other sources of information such as acoustic and textual (transcript) representation of the spoken languages are also incorporated in the analysis. Multi-modal analysis (e.g. sentiment analysis Zadeh et al. 2018c, emotion recognition Poria et al. 2016, question-answering Teney et al. 2017 etc.) is an emerging field of study, that uti- lizes multiple information sources for solving a problem. These sources (e.g., text, visual, acous- tic, etc.) offer a diverse and often distinct piece of information that a system can leverage on. For ex- ample, ‘text’ carries semantic information of the spoken sentence, whereas ‘acoustic’ information reveals the emphasis (pitch, voice quality) on each word. In contrast, the ‘visual’ information (image or video frame) extracts the gesture and posture of the speaker. Traditionally, ‘text’ has been the key factor in any Natural Language Processing (NLP) tasks in- cluding sentiment and emotion analysis. However, with the recent emergence of social media plat- forms and their available multi-modal contents, an interdisciplinary study involving text, acoustic and visual features have drawn significant interest among the research community. …

Content truncated to 500 words