Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 269–280 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 269 Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition Wenliang Dai, Zihan Liu, Tiezheng Yu, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {wdaiai,zliucr,tyuah}@connect.ust.hk, pascale@ece.ust.hk Abstract Despite the recent achievements made in the multi-modal emotion recognition task, two problems still exist and have not been well in- vestigated: 1) the relationship between differ- ent emotion categories are not utilized, which leads to sub-optimal performance; and 2) cur- rent models fail to cope well with low-resource emotions, especially for unseen emotions. In this paper, we propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. We use pre-trained word embeddings to represent emotion cate- gories for textual data. Then, two mapping functions are learned to transfer these embed- dings into visual and acoustic spaces. For each modality, the model calculates the representa- tion distance between the input sequence and target emotions and makes predictions based on the distances. By doing so, our model can directly adapt to the unseen emotions in any modality since we have their pre-trained em- beddings and modality mapping functions. Ex- periments show that our model achieves state- of-the-art performance on most of the emotion categories. Besides, our model also outper- forms existing baselines in the zero-shot and few-shot scenarios for unseen emotions 1. 1 Introduction Multi-modal emotion recognition is an increasingly popular but challenging task. One main challenge is that labelled data is difficult to come by as hu- mans find it time-consuming to discern emotion categories from either speech or video. Indeed we humans express emotions through a combination of modalities, including the way we speak, the words we use, facial expressions and sometimes gestures. It is also much more comfortable for humans to un- derstand each other’s emotions when they can both 1Code is available at https://github.com/ wenliangdai/Modality-Transferable-MER happy surprised Acoustic�Space happy surprised Textual�Space happy surprised Visual�Space He�proposed.��Well,�and�I�said�yes,�of�course. Textual Visual Acoustic I'm�going�to�miss�you. Textual Visual Acoustic .�.�. sad sad sad (happy�and� surprised) (sad) Seen�Emotion�Example Unseen�Emotion�Example .�.�. Figure 1: An intuitive example of our method. In the upper image, the relative positions of GloVe emotion embeddings (happy, surprised) are shown in the tex- tual space, which are then projected to acoustic and vi- sual spaces by two mapping functions (ft→a and ft→v). Our model learns to group the representations of input sentences ( , ) based on their corresponding emotion embeddings. Examples are shown in the lower image. When a sample has both happy and surprised emotions, its representation gets close to these two emotion em- beddings in all three spaces. If an unseen emotion sad ( ) comes, the model processes it with the same flow and recognizes corresponding data samples. hear and see the other person. …

Content truncated to 500 words