Emotional Perception of Fairy Tales: Achieving Agreement in Emotion Annotation of Text

Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 98–106, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Emotional Perception of Fairy Tales: Achieving Agreement in Emotion Annotation of Text Ekaterina P. Volkova1,2, Betty J. Mohler2, Detmar Meurers1, Dale Gerdemann1, Heinrich H. B¨ulthoff2 1 Universit¨at T¨ubingen, Seminar f¨ur Sprachwissenschaft 19 Wilchelmstr., T¨ubingen, 72074, Germany 2 Max Planck Institute for Biological Cybernetics 38 Spemannstr., T¨ubingen, 72076, Germany Abstract Emotion analysis (EA) is a rapidly developing area in computational linguistics. An EA system can be extremely useful in fields such as information retrieval and emotion-driven computer animation. For most EA systems, the number of emotion classes is very limited and the text units the classes are assigned to are discrete and predefined. The question we address in this paper is whether the set of emotion categories can be enriched and whether the units to which the categories are assigned can be more flexibly defined. We present an experiment showing how an annotation task can be set up so that untrained participants can perform emotion analysis with high agreement even when not restricted to a predetermined annotation unit and using a rich set of emotion categories. As such it sets the stage for the development of more complex EA systems which are closer to the actual human emotional perception of text. 1 Introduction As a first step towards developing an emotion analysis (EA) system simulating human emotional perception of text, it is important to research the nature of the emotion analysis performed by humans and examine whether they can reliably perform the task. To investigate these issues, we conducted an experiment to find out the strategies people use to annotate selected folk fairy tale texts for emotions. The participants had to choose from a set of fifteen emotion categories, a significantly larger set than typically used in EA, and assign them to an unrestricted range of text. To explore whether human annotators can reliably perform a task, inter-annotator agreement (IAA) (Artstein and Poesio, 2008) is the relevant measure. This measure can be calculated between every two individual annotations in order to find pairs or even teams of annotators whose strategies seem to be consistent and coherent enough so that they can be used further as the gold-standard annotation suited to train a machine learning approach for automatic EA analysis. A resulting EA system, capable of simulating human emotional perception of text, would be useful for information retrieval and many other fields. There are two main aspects of the resulting anno- tations to be researched. First, how consistently can people perceive and locate the emotional aspect of fairy tale texts? Second, how do they express their perception of text by means of annotation strategies? In the next sections, we address these questions and provide details of an experiment we conducted to empirically advance our understanding of the issues. 2 Motivation and Aimed Application Most existing EA systems are implemented for and used …

Content truncated to 500 words