When to Laugh and How Hard? A Multimodal Approach to Detecting Humor and Its Intensity

Proceedings of the 29th International Conference on Computational Linguistics, pages 6875–6886 October 12–17, 2022. 6875 When to Laugh and How Hard? A Multimodal Approach to Detecting Humor and its Intensity Khalid Alnajjar1,2, Mika Hämäläinen1,2 Jörg Tiedemann1, Jorma Laaksonen3 and Mikko Kurimo3 1University of Helsinki, Finland 2Rootroo Oy, Finland 3Aalto University, Finland firstname.lastname@{helsinki.fi1,aalto.fi3} Abstract Prerecorded laughter accompanying dialog in comedy TV shows encourages the audience to laugh by clearly marking humorous moments in the show. We present an approach for auto- matically detecting humor in the Friends TV show using multimodal data. Our model is ca- pable of recognizing whether an utterance is hu- morous or not and assess the intensity of it. We use the prerecorded laughter in the show as an- notation as it marks humor and the length of the audience’s laughter tells us how funny a given joke is. We evaluate the model on episodes the model has not been exposed to during the training phase. Our results show that the model is capable of correctly detecting whether an ut- terance is humorous 78% of the time and how long the audience’s laughter reaction should last with a mean absolute error of 600 millisec- onds. 1 Introduction Humor is a topic that has piqued interest of the computational creativity research community over the years. There are numerous systems that can generate humor using a variety of different meth- ods (Weller et al., 2020; Tyler et al., 2020; Alnajjar and Hämäläinen, 2021b). But just as important as it is to research generation from the computa- tional creativity perspective, it is to study automatic assessment of humor. The role of humor is an important one for us humans as it has it’s own social function (Ziv, 2010). It helps us talk about difficult topics (Vivona, 2013; Monahan, 2015) and relieves tension (Shur- cliff, 1968). Laughter has a role in building rela- tionships (McCabe et al., 2017; Kurtz and Algoe, 2017) and it has a positive effect on brain chem- istry (Gonot-Schoupinsky and Garip, 2018). Hu- mor and laughter are therefore an integral part of who we are as a species. Humor is a phenomenon that requires sur- prise and coherence (see Hämäläinen and Alna- jjar (2019)); or incongruity and its resolution in other terms (Raskin, 1985; Attardo and Raskin, 1991). However, what is surprising or incongruous, depends heavily on the context where humour is presented. Quite indeed, something intended as a joke can be seen as a severe insult just by a change in context. Therefore, we believe that a multimodal approach needs to be researched when when as- sessing humor automatically; something that thus far has been researched by focusing on the textual modality alone. Annotated multimodal datasets are scarce, but an access to such a dataset is crucial for any computa- tional attempt on humor detection and assessment. For this reason, we embrace a clever approach: we use episodes of the beloved American sitcom Friends as our data source. The TV show has pre- recorded audience laughter …

Content truncated to 500 words