Utterance-Level Multimodal Sentiment Analysis

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 973–982, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Utterance-Level Multimodal Sentiment Analysis Ver´onica P´erez-Rosas and Rada Mihalcea Computer Science and Engineering University of North Texas veronicaperezrosas@my.unt.edu, rada@cs.unt.edu Louis-Philippe Morency Institute for Creative Technologies University of Southern California morency@ict.usc.edu Abstract During real-life interactions, people are naturally gesturing and modulating their voice to emphasize specific points or to express their emotions. With the recent growth of social websites such as YouTube, Facebook, and Amazon, video reviews are emerging as a new source of multimodal and natural opinions that has been left al- most untapped by automatic opinion anal- ysis techniques. This paper presents a method for multimodal sentiment classi- fication, which can identify the sentiment expressed in utterance-level visual datas- treams. Using a new multimodal dataset consisting of sentiment annotated utter- ances extracted from video reviews, we show that multimodal sentiment analysis can be effectively performed, and that the joint use of visual, acoustic, and linguistic modalities can lead to error rate reductions of up to 10.5% as compared to the best performing individual modality. 1 Introduction Video reviews represent a growing source of con- sumer information that gained increasing interest from companies, researchers, and consumers. Pop- ular web platforms such as YouTube, Amazon, Facebook, and ExpoTV have reported a signifi- cant increase in the number of consumer reviews in video format over the past five years. Compared to traditional text reviews, video reviews provide a more natural experience as they allow the viewer to better sense the reviewer’s emotions, beliefs, and intentions through richer channels such as intona- tions, facial expressions, and body language. Much of the work to date on opinion analysis has focused on textual data, and a number of resources have been created including lexicons (Wiebe and Riloff, 2005; Esuli and Sebastiani, 2006) or large annotated datasets (Maas et al., 2011). Given the accelerated growth of other media on the Web and elsewhere, which includes massive collections of videos (e.g., YouTube, Vimeo, VideoLectures), im- ages (e.g., Flickr, Picasa), audio clips (e.g., pod- casts), the ability to address the identification of opinions in the presence of diverse modalities is be- coming increasingly important. This has motivated researchers to start exploring multimodal clues for the detection of sentiment and emotions in video content (Morency et al., 2011; Wagner et al., 2011). In this paper, we explore the addition of speech and visual modalities to text analysis in order to identify the sentiment expressed in video reviews. Given the non homogeneous nature of full-video reviews, which typically include a mixture of posi- tive, negative, and neutral statements, we decided to perform our experiments and analyses at the ut- terance level. This is in line with earlier work on text-based sentiment analysis, where it has been observed that full-document reviews often contain both positive and negative comments, which led to a number of methods addressing opinion analysis at sentence level. Our results show that relying on …

Content truncated to 500 words