A Sentiment and Emotion Aware Multimodal Multiparty Humor Recognition in Multilingual Conversational Setting

Proceedings of the 29th International Conference on Computational Linguistics, pages 6752–6761 October 12–17, 2022. 6752 A Sentiment and Emotion aware Multimodal Multiparty Humor Recognition in Multilingual Conversational Setting Dushyant Singh Chauhan1∗, Gopendra Vikram Singh1, Aseem Arora1, Asif Ekbal1, and Pushpak Bhattacharyya2 Department of Computer Science and Engineering 1 Indian Institute of Technology Patna, India {1821cs17,gopendra_1921cs15,aseem_1911mc02,asif}@iitp.ac.in 2 Indian Institute of Technology Bombay, India pb@cse.iitb.ac.in Abstract In this paper, we hypothesize that humor is closely related to sentiment and emotions. Also, due to the tremendous growth in multilingual content, there is a great demand for building models and systems that support multilingual information access. To this end, we first ex- tend the recently released Multimodal Multi- party Hindi Humor (M2H2) dataset by adding parallel English utterances corresponding to Hindi utterances and then annotating each utter- ance with sentiment and emotion classes. We name it Sentiment, Humor, and Emotion aware Multilingual Multimodal Multiparty Dataset (SHEMuD). Therefore, we propose a multi- task framework wherein the primary task is humor detection, and the auxiliary tasks are sentiment and emotion identification. We de- sign a multitasking framework wherein we first propose a Context Transformer to capture the deep contextual relationships with the input ut- terances. We then propose a Sentiment and Emotion aware Embedding (SE-Embedding) to get the overall representation of a particular emotion and sentiment w.r.t. the specific humor situation. Experimental results on the SHEMuD show the efficacy of our approach and shows that multitask learning offers an improvement over the single-task framework for both mono- lingual (4.86 points ↑ in Hindi and 5.9 points ↑ in English in F1-score) and multilingual (5.17 points ↑ in F1-score) setting. 1 Introduction Humor (Ritschel et al., 2019, 2020; Song et al., 2021; Chauhan et al., 2021) is an essential aspect of daily conversation, and people try to provoke hu- mor in their talks. Warren et al. (2018) defined hu- mor as "the nature of experiences to induce laugh- ter and provide amusement". Humor is a tool by which anyone can draw the audience’s attention. Even in a formal conversation, humor may make a ∗The first two authors contributed equally to this work and are jointly the first authors. person look more attractive and thus may lead to a better conclusion. Irrespective of its relation to intelligence, hu- mor is inherently a challenging problem to under- stand. To understand humor, we also take some additional information into consideration i.e., sen- timent (Ghosal et al., 2018; Chauhan et al., 2019, 2020a, 2022) and emotion (Russell and Barrett, 1999; Pagé Fortin and Chaib-draa, 2019; Chauhan et al., 2020b) to help humor detection in conver- sations. Sentiment and emotion are two aspects that help each other, which has already been shown in (Akhtar et al., 2019). We hypothesize that sen- timent and emotion do not only help each other (Akhtar et al., 2019) but also help humor. For ex- ample, it is difficult to decide whether the following utterance "a�� �� ��� ��� ��� �� ��� �� � � �� ����। …

Content truncated to 500 words