RED v2: Enhancing RED Dataset for Multi-Label Emotion Detection

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1392–1399 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1392 RED v2: Enhancing RED Dataset for Multi-Label Emotion Detection Alexandra Ciobotaru1, Mihai V. Constantinescu2 Liviu P. Dinu1, Stefan Daniel Dumitrescu3 University of Bucharest, Faculty of Mathematics and Computer Science1, Independent Researcher2 Adobe3 Bucharest, Romania alexandra.ciobotaru@unibuc.ro, mihai.vlad6@gmail.com ldinu@fmi.unibuc.ro, sdumitre@adobe.com Abstract RED (Romanian Emotion Dataset) is a machine learning-based resource developed for the automatic detection of emotions in Romanian texts, containing single-label annotated tweets with one of the following emotions: joy, fear, sadness, anger and neutral. In this work, we propose REDv2, an open-source extension of RED by adding two more emotions, trust and surprise, and by widening the annotation schema so that the resulted novel dataset is multi-label. We show the overall reliability of our dataset by computing inter-annotator agreements per tweet using a formula suitable for our annotation setup and we aggregate all annotators’ opinions into two variants of ground truth, one suitable for multi-label classification and the other suitable for text regression. We propose strong baselines with two transformer models, the Romanian BERT and the multilingual XLM-Roberta model, in both categorical and regression settings. Keywords: emotion detection, multi-label classification, text regression, Romanian tweets 1. Introduction Interpreting correctly one’s own emotions, as well as other people’s emotional states, is a central aspect of emotional intelligence. Today, people can automate the process of emotion detection by creating machine learning models, provided by the fact that the model training was done on qualitative and sufficient data. With the constant increase of social media usage there is also an increase in online public data, freely available for model creation. Thus, analyzing emotions in online content naturally has became more and more of a topic of interest in the recent years. Sentiment analysis, along with emotion analysis, are key technologies used to gain insights from social me- dia networks, the opinion mining field reaching a level of maturity, sufficient enough to be used in practical applications (Iglesias and Moreno, 2019). There are many definitions to emotions (Peng et al., 2021), but as defined in (Liu, 2015), an emotion is “a mental state that arises spontaneously rather than through a conscious effort; it is also often accompanied by physiological changes.”. Many psychologists tried to create a taxonomy of emotions, but probably one of the most agreed upon taxonomy is Robert Plutchick’s wheel of emotions (Plutchik, 1973). He identified eight primary emotions: happiness, trust, fear, surprise, sad- ness, disgust, anger and anticipation. Having a model that automatically detects emotions in text has a wide range of applications, from comput- ing the overall opinion of clients and/or potential cus- tomers in the field of brand management (Istrati and Ciobotaru, 2022), to automatic adaptation of chatbot answers in respect to the user’s emotional state. The first dataset of single labeled texts for detecting emotions from Romanian content is REDv1 (Roma- nian Emotion Dataset) (Ciobotaru and Dinu, 2021), a dataset containing …

Content truncated to 500 words