CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1801–1812, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1801 CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French Amir Zadeh1, Yan Sheng Cao2, Simon Hessner1, Paul Pu Liang3, Soujanya Poria4, Louis-Philippe Morency1 (1) LTI, SCS, CMU, (2) SCS, CMU, (3) MLD, SCS, CMU (4) Singapore University of Technology and Design {abagherz,yanshenc,shessner,pliang,morency}@cs.cmu.edu sporia@sutd.edu.sg Access link: https://bit.ly/2Svbg9f e nós vamos ler esse livro meio as cegas para no fim de julho mesmo vim conversar com vocês sobre a história Start=1:38.49 End=1:44.08 Sentence #20 Video Start=0:00.00 Video End=2:41.86 Facial Action Units Shape Parameters Facial Landmarks Head Pose Gaze HOG Features Sentence #21 Sentence #19 … … Sentence #30 Sentence #0 Visual Modality Language Modality Acoustic Modality NAQ MFCC Peak Slope VUV & SP MDP GSP Phoneme Alignment Sentiment (1 × L) Subjectivity (1 × B) Emotions (6 × L) Attributes (12 × B) Sentence-level Labels: … … Figure 1: Overview of in-the-wild monologue videos and sentence utterances in the CMU-MOSEAS dataset. Each sentence is annotated for 20 labels including sentiment, subjectivity, emotions and attributes. “L” denotes Likert (intensity) and “B” denotes Binary for the type of the labels. The example above is a Portuguese video. Abstract Modeling multimodal language is a core re- search area in natural language processing. While languages such as English have rela- tively large multimodal language resources, other widely spoken languages across the globe have few or no large-scale datasets in this area. This disproportionately affects na- tive speakers of languages other than English. As a step towards building more equitable and inclusive multimodal systems, we intro- duce the first large-scale multimodal language dataset for Spanish, Portuguese, German and French. The proposed dataset, called CMU- MOSEAS (CMU Multimodal Opinion Senti- ment, Emotions and Attributes), is the largest of its kind with 40,000 total labelled sen- tences. It covers a diverse set topics and speak- ers, and carries supervision of 20 labels includ- ing sentiment (and subjectivity), emotions, and attributes. Our evaluations on a state- of-the-art multimodal model demonstrates that CMU-MOSEAS enables further research for multilingual studies in multimodal language. 1 Introduction Humans use a coordinated multimodal signal to communicate with each other. This communica- tion signal is called multimodal language (Perniss, 2018); a complex temporal and idiosyncratic sig- nal which includes the modalities of language, visual and acoustic. On a daily basis across the world, intentions and emotions are conveyed through joint utilization of these three modali- ties. While English, Chinese, and Spanish lan- guages have resources for computational analy- sis of multimodal language (focusing on analysis of sentiment, subjectivity, or emotions (Yu et al., 2020; Poria et al., 2019; Zadeh et al., 2018b; Park et al., 2014; W¨ollmer et al., 2013; Poria et al., 2020)), other commonly spoken languages across the globe lag behind. As Artificial Intelligence 1802 (AI) increasingly blends into everyday life across the globe, there is a genuine need for intelligent entities capable of understanding multimodal lan- guage in …

Content truncated to 500 words