Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2539–2544, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-Level Multimodal Sentiment Analysis Soujanya Poria Temasek Laboratory, Nanyang Technological University, Singapore sporia@ntu.edu.sg Erik Cambria School of Computer Engineering, Nanyang Technological University, Singapore cambria@ntu.edu.sg Alexander Gelbukh Centro de Investigación en Computación, Instituto Politécnico Nacional, Mexico www.gelbukh.com Abstract We present a novel way of extracting fea- tures from short texts, based on the acti- vation values of an inner layer of a deep convolutional neural network. We use the extracted features in multimodal senti- ment analysis of short video clips repre- senting one sentence each. We use the combined feature vectors of textual, vis- ual, and audio modalities to train a classi- fier based on multiple kernel learning, which is known to be good at heteroge- neous data. We obtain 14% performance improvement over the state of the art and present a parallelizable decision-level da- ta fusion method, which is much faster, though slightly less accurate. 1 Introduction The advent of the Social Web has enabled any- one with a smartphone or computer to easily cre- ate and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is video. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of vid- eo on the Internet will only continue to increase. It has become increasingly difficult for re- searchers to keep up with this deluge of video content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis (Cambria et al., 2013a; 2013b; 2014), as both service and product reviews are gradually shifting from unimodal to multimodal. We present a method for detecting sentiment polarity in short video clips of a person uttering a sentence. We do it using all three modalities: visual, such as facial expression, audio, such as pitch, and textual, the contents of the uttered sentence. While the visual and the audio modalities pro- vide additional evidence that improves classifica- tion accuracy, we found the textual modality to have the greater impact on the result (Cambria and Hussain, 2015; Cambria et al., 2013c; Poria et al., 2015a; 2015b). In this paper, we propose a novel way for fea- ture extraction from text. Given a training corpus with hand-annotated sentiment polarity labels, following Kim (2014), we train a deep convolu- tional neural network (CNN) on it. However, instead of using it as a classifier, as Kim did, we use the values from its hidden layer as features for a much more advanced classifier, which gives superior accuracy. Similar ideas have been sug- gested in the context of computer …

Content truncated to 500 words