COMMA-DEER: COmmon-sense Aware Multimodal Multitask Approach for Detection of Emotion and Emotional Reasoning in Conversations

Proceedings of the 29th International Conference on Computational Linguistics, pages 6978–6990 October 12–17, 2022. 6978 COMMA-DEER: COmmon-sense aware Multimodal Multitask Approach for Detection of Emotion and Emotional Reasoning in Conversations Soumitra Ghosh1*, Gopendra Vikram Singh1*, Asif Ekbal1 and Pushpak Bhattacharyya2 1Department of Computer Science and Engineering, IIT Patna, India 2Department of Computer Science and Engineering, IIT Bombay, India ghosh.soumitra2@gmail.com, {gopendra_1921cs15,asif}@iitp.ac.in, pb@cse.iitb.ac.in ∗ Abstract Mental health is a critical component of the United Nations’ Sustainable Development Goals (SDGs), particularly Goal 3, which aims to provide "good health and well-being". The present mental health treatment gap is exacer- bated by stigma, lack of human resources, and lack of research capability for implementation and policy reform. We present and discuss a novel task of detecting emotional reasoning (ER) and accompanying emotions in conversa- tions. In particular, we create a first-of-its-kind multimodal mental health conversational cor- pus that is manually annotated at the utterance level with emotional reasoning and related emo- tion. We develop a multimodal multitask frame- work with a novel multimodal feature fusion technique and a contextuality learning module to handle the two tasks. Leveraging multimodal sources of information, commonsense reason- ing, and through a multitask framework, our proposed model produces strong results. We achieve performance gains of 6% accuracy and 4.62% F1 on the emotion detection task and 3.56% accuracy and 3.31% F1 on the ER de- tection task, when compared to the existing state-of-the-art model. 1 Introduction Each year, mental illness costs the economy almost 12 billion working days. Mental diseases are pre- dicted to cost the global economy $16 trillion by 2030 (Canady, 2018). Despite extensive evidence of the close association between mental health and nearly every major issue in development, people with mental illnesses are among those most at risk of being left out of development programmes. Pro- moting mental health and preventing mental illness should be part of larger efforts to meet the United Nations’ Sustainable Development Goals. Unlike object identification tasks, emotion per- ception is significantly impacted by personal bias, ∗* The first two authors contributed equally to this work and are jointly the first authors. cultural backgrounds, and contextual information (such as the environment). Emotional information can function as an authoritative inner voice in those who suffer from anxiety, despair, guilt, or fear on a daily basis (Caprara and Cervone, 2000). Nor- mal individuals can suppress or remove the mood influence, while clinical patients cannot stop the inner voice from being viewed as authoritative. In spite of contradictory empirical facts, someone may draw the conclusion that something is true based only on their emotional response. This process is known as emotional reasoning (Gangemi et al., 2013). For example, • I feel insecure about my wife, which means she must be cheating on me. • I feel afraid, so my neighbors must have been spying on me. While automatic emotion recognition in conver- sations can facilitate developing intelligent systems, such as empathetic chatbots, customers’ feedback assessment, etc., automatic identification of emo- tional reasoning from conversations can …

Content truncated to 500 words