WRIME: A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2095–2104 June 6–11, 2021. ©2021 Association for Computational Linguistics 2095 WRIME: A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations Tomoyuki Kajiwara Graduate School of Science and Engineering Ehime University, Japan kajiwara@cs.ehime-u.ac.jp Chenhui Chu Graduate School of Informatics Kyoto University, Japan chu@i.kyoto-u.ac.jp Noriko Takemura Yuta Nakashima Institute for Datability Science, Osaka University, Japan {takemura, n-yuta, nagahara}@ids.osaka-u.ac.jp Hajime Nagahara Abstract We annotate 17,000 SNS posts with both the writer’s subjective emotional intensity and the reader’s objective one to construct a Japanese emotion analysis dataset. In this study, we ex- plore the difference between the emotional in- tensity of the writer and that of the readers with this dataset. We found that the reader cannot fully detect the emotions of the writer, espe- cially anger and trust. In addition, experimen- tal results in estimating the emotional inten- sity show that it is more difficult to estimate the writer’s subjective labels than the readers’. The large gap between the subjective and ob- jective emotions implies the complexity of the mapping from a post to the subjective emo- tional intensities, which also leads to a lower performance with machine learning models. 1 Introduction Emotion analysis is one of the major NLP tasks with a wide range of applications, such as a dia- logue system (Tokuhisa et al., 2008) and social me- dia mining (Stieglitz and Dang-Xuan, 2013). Since emotion analysis has been actively studied, not only the classification of the sentiment polarity (posi- tive or negative) of the text (Socher et al., 2013), but also more detailed emotion detection and emo- tional intensity estimation (Bostan and Klinger, 2018) have been attempted in recent years. Pre- vious studies on emotion analysis use six emotions (anger, disgust, fear, joy, sadness, and surprise) by Ekman (1992), eight emotions (anger, disgust, fear, joy, sadness, surprise, trust, and anticipa- tion) by Plutchik (1980), and VAD model (Valence, Arousal, and Dominance) by Russell (1980). Table 1 lists datasets with emotional intensity.1 1In this paper, the emotions of the text writers themselves are called subjective emotions, and the emotions that the read- ers receive from the text are called objective emotions. These existing emotion analysis datasets include subjective emotional intensity labels by the writ- ers (Scherer and Wallbott, 1994) and objective ones by the readers (Aman and Szpakowicz, 2007; Strap- parava and Mihalcea, 2007; Buechel and Hahn, 2017; Mohammad and Bravo-Marquez, 2017a; Mohammad and Kiritchenko, 2018; Bostan et al., 2020), whereas the latter is mainly done by, e.g., expert or crowdsourcing annotators. It depends on the applications whether the writer’s emotions or the reader’s ones to be esti- mated in NLP-based emotion analysis. For exam- ple, in a dialogue system, it is important to estimate the reader’s emotion because we want to know how the user feels in response to the system’s utterance. On the other hand, in applications such as social media mining, we want to estimate …

Content truncated to 500 words