UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2046–2056, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2046 UR-FUNNY: A Multimodal Language Dataset for Understanding Humor Md Kamrul Hasan1*, Wasifur Rahman1*, Amir Zadeh2, Jianyuan Zhong1, Md Iftekhar Tanveer1 †, Louis-Philippe Morency2, Mohammed (Ehsan) Hoque1 1 - Department of Computer Science, University of Rochester, USA 2 - Language Technologies Institute, SCS, CMU, USA {mhasan8,echowdh2}@cs.rochester.edu, abagherz@cs.cmu.edu, jzhong9@u.rochester.edu, itanveer@cs.rochester.edu, morency@cs.cmu.edu, mehoque@cs.rochester.edu I can put a bizarre new idea in your mind right now. I can say, imagine a jellyfish waltzing in a library while thinking about quantum mechanics. Now, if everything has gone relatively well in your life, you probably haven’t had that thought before. So because of this ability, we humans are able to transmit knowledge across time. Your brain takes those vibrations from your eardrums and transforms them into thoughts. Video Audio Text Humor? Yes No Context Punchline t=0:41 t=1:15 t=1:20 t=1:09 t=1:03 t=0:52 Figure 1: An example of the UR-FUNNY dataset. UR-FUNNY presents a framework to study the dynamics of humor in multimodal language. Machine learning models are given a sequence of sentences with the accompanying modalities of visual and acoustic. Their goal is to detect whether or not the sequence will trigger immediate laughter after the punchline. Abstract Humor is a unique and creative communica- tive behavior often displayed during social in- teractions. It is produced in a multimodal man- ner, through the usage of words (text), gestures (visual) and prosodic cues (acoustic). Un- derstanding humor from these three modali- ties falls within boundaries of multimodal lan- guage; a recent research trend in natural lan- guage processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal con- text it has been understudied. This paper presents a diverse multimodal dataset, called UR-FUNNY, to open the door to understand- ing multimodal language used in expressing humor. The dataset and accompanying stud- ies, present a framework in multimodal humor detection for the natural language processing community. UR-FUNNY is publicly available for research. 1 Introduction Humor is a unique communication skill that re- moves barriers in conversations. Research shows that effective use of humor allows a speaker to establish rapport (Stauffer, 1999), grab atten- tion (Wanzer et al., 2010), introduce a difficult concept without confusing the audience (Garner, 2005) and even to build trust (Vartabedian and Vartabedian, 1993). During face-to-face com- munications, humor involves multimodal com- municative channels including effective use of words (text), accompanying gestures (visual) and prosodic cues (acoustic). Being able to mix and align those modalities appropriately is often unique to individuals, attributing to many differ- ent styles. Styles include gradually building up to a punchline using words, gestures and prosodic cues, a sudden twist to the story with an unex- * - Equal contribution ; †- Currently affiliated with Comcast Applied AI Research, Washington DC, USA. 2047 …

Content truncated to 500 words