A Multimodal Corpus for Emotion Recognition in Sarcasm

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6992–7003 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6992 A Multimodal Corpus for Emotion Recognition in Sarcasm Anupama Ray, Shubham Mishra, Apoorva Nunna, Pushpak Bhattacharyya IBM Research India, Department of Computer Science and Engineering, IIT Bombay, India anupamar@in.ibm.com, shubham101mishra@gmail.com, {apoorvanunna, pb}@cse.iitb.ac.in Abstract While sentiment and emotion analysis have been studied extensively, the relationship between sarcasm and emotion has largely remained unexplored. A sarcastic expression may have a variety of underlying emotions. For example, “I love being ignored” belies sadness, while “my mobile is fabulous with a battery backup of only 15 minutes!” expresses frustration. Detecting the emotion behind a sarcastic expression is non-trivial yet an important task. We undertake the task of detecting the emotion in a sarcastic statement, which to the best of our knowledge, is hitherto unexplored. We start with the recently released multimodal sarcasm detection dataset (MUStARD) pre-annotated with 9 emotions. We identify and correct 343 incorrect emotion labels (out of 690). We double the size of the dataset, label it with emotions along with valence and arousal which are important indicators of emotional intensity. Finally, we label each sarcastic utterance with one of the four sarcasm types-Propositional, Embedded, Likeprefixed and Illocutionary, with the goal of advancing sarcasm detection research. Exhaustive experimentation with multimodal (text, audio, and video) fusion models establishes a benchmark for exact emotion recognition in sarcasm and outperforms the state-of-art sarcasm detection. We release the dataset enriched with var- ious annotations and the code for research purposes: https://github.com/apoorva-nunna/MUStARD_Plus_Plus Keywords: Emotion understanding, sarcasm, multimodal, valence-arousal 1. Introduction Emotion understanding leads to a deeper insight into the intent of the speaker and is key to generating the right response in conversational systems. Detecting emotions and sarcasm is crucial for all services involving human interactions, such as chatbots, e-commerce, e-tourism, and several other businesses. To be able to understand the user’s intent, we started with the research problem on understanding the emotions that lead to the usage of sarcasm in a conversation. Sarcasm is a very sophisti- cated linguistic articulation where the surface meaning often stands in contrast to the underlying deeper mean- ing. While this incongruity is the key element of sar- casm, the intent could be to appear humorous, ridicule someone, or to express contempt. Thus sarcasm is con- sidered a very nuanced or intelligent language construct that poses several challenges to emotion recognition; for example, perceived emotion could be completely flipped due to the presence of sarcasm. Sarcasm often relies on verbal and non-verbal cues (pitch, tone, em- phasis in speech, and body language in video). Even for humans, understanding the underlying emotion is challenging without the audio/video or the context of the conversation. However, researchers have worked on sarcasm detection on text modality with textual datasets (such as tweets (Oprea and Magdy, 2020), Reddit short texts (Khodak et al., 2017), dialogue (Oraby et al., 2016) etc) for a decade. Recently we have seen multimodal …

Content truncated to 500 words