Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction

Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction Enrica Troiano∗ Institut f¨ur Maschinelle Sprachverarbeitung University of Stuttgart enrica.troiano@ims.uni -stuttgart.de Laura Oberl¨ander∗ Institut f¨ur Maschinelle Sprachverarbeitung University of Stuttgart laura.oberlaender@ims.uni -stuttgart.de Roman Klinger∗ Institut f¨ur Maschinelle Sprachverarbeitung University of Stuttgart roman.klinger@ims.uni-stuttgart.de The most prominent tasks in emotion analysis are to assign emotions to texts and to understand how emotions manifest in language. An important observation for natural language processing is that emotions can be communicated implicitly by referring to events alone, appealing to an empathetic, intersubjective understanding of events, even without explicitly mentioning an emotion name. In psychology, the class of emotion theories known as appraisal theories aims at explaining the link between events and emotions. Appraisals can be formalized as variables that measure a cognitive evaluation by people living through an event that they consider relevant. They include the assessment if an event is novel, if the person considers themselves to be responsible, if it is in line with their own goals, and so forth. Such appraisals explain which emotions are developed based on an event, for example, that a novel situation can induce surprise or one with uncertain consequences could evoke fear. We analyze the suitability of appraisal ∗ All authors contributed equally. Action Editor: Saif M. Mohammad. Submission received: 10 June 2022; revised version received: 25 July 2022; accepted for publication: 24 August 2022. https://doi.org/10.1162/coli a 00461 © 2022 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume 49, Number 1 theories for emotion analysis in text with the goal of understanding if appraisal concepts can reliably be reconstructed by annotators, if they can be predicted by text classifiers, and if appraisal concepts help to identify emotion categories. To achieve that, we compile a corpus by asking people to textually describe events that triggered particular emotions and to disclose their appraisals. Then, we ask readers to reconstruct emotions and appraisals from the text. This set-up allows us to measure if emotions and appraisals can be recovered purely from text and provides a human baseline to judge a model’s performance measures. Our comparison of text classification methods to human annotators shows that both can reliably detect emotions and appraisals with similar performance. Therefore, appraisals constitute an alternative computational emotion analysis paradigm and further improve the categorization of emotions in text with joint models. 1. Introduction Voices that have had a say about the affective life of humans have been raised from multiple disciplines. Over the centuries, philosophers, neuroscientists, and cognitive and computational researchers have been drawn to the study of passions, feelings, and sentiment (Scarantino 2016; Adolphs 2017; Oatley and Johnson-Laird 2014; Karg et al. 2013). Among such affective phenomena, emotions stand out. For one thing, they are many: While sentiment can be described with a handful of categories (e.g., neutral, negative, positive), it takes a varied vocabulary to distinguish the mental state that accompanies a cheerful laughter from that enticing a …

Content truncated to 500 words