Multimodal and Multi-view Models for Emotion Recognition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 991–1002 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 991 Multimodal and Multi-view Models for Emotion Recognition Gustavo Aguilar‡, Viktor Rozgi´c†, Weiran Wang† and Chao Wang† University of Houston‡ Amazon.com† gaguilaralas@uh.edu, {rozgicv, weiranw, wngcha}@amazon.com Abstract Studies on emotion recognition (ER) show that combining lexical and acoustic information re- sults in more robust and accurate models. The majority of the studies focus on settings where both modalities are available in training and evaluation. However, in practice, this is not always the case; getting ASR output may rep- resent a bottleneck in a deployment pipeline due to computational complexity or privacy- related constraints. To address this challenge, we study the problem of efficiently combining acoustic and lexical modalities during train- ing while still providing a deployable acous- tic model that does not require lexical inputs. We first experiment with multimodal models and two attention mechanisms to assess the extent of the benefits that lexical information can provide. Then, we frame the task as a multi-view learning problem to induce se- mantic information from a multimodal model into our acoustic-only network using a con- trastive loss function. Our multimodal model outperforms the previous state of the art on the USC-IEMOCAP dataset reported on lex- ical and acoustic information. Additionally, our multi-view-trained acoustic network sig- nificantly surpasses models that have been ex- clusively trained with acoustic features. 1 Introduction The task of emotion recognition (ER) requires understanding the way humans interact to ex- press their emotional state during conversations. Among others, emotions are encoded in both lexi- cal and acoustic information where each modality contributes to the overall emotional state of a given speaker. However, in some situations, one modal- ity can be more insightful to derive emotions than the other. For instance, the phrase “yeah... course” does not have enough lexical informationof to derive the right emotion, and it may all depend on the acoustic patterns. On the other hand, the phrase “I really miss my dog!” does not need acoustic information to detect that the most likely emotion is sadness. Thus, recognizing emotions is not a trivial task because an emotional state can be easily shaped by many factors: context, word content, spectral and prosodic information, among others (Barbulescu et al., 2017). In this paper, we study the emotion recognition problem from the speech and language perspec- tives. We formally look into acoustic and lexi- cal modalities with the aim of improving models that only use acoustic information. In the first part of this work, our goal is to assess the extent to which lexical information benefits acoustic mod- els. We propose a multimodal method that is in- spired by the way humans process emotions in a conversation. That is, lexical and acoustic infor- mation is simultaneously perceived at every word step. Hence, we introduce the concept of acous- tic words: word-level representations derived from acoustic features in a speech fragment. The acous- tic …

Content truncated to 500 words