Evaluation of Unsupervised Emotion Models to Textual Affect Recognition

Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 62–70, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Evaluation of Unsupervised Emotion Models to Textual Affect Recognition Sunghwan Mac Kim School of Electrical and Information Engineering University of Sydney Sydney, Australia skim1871@uni.sydney.edu.au Alessandro Valitutti Department of Cognitive Science and Education University of Trento Trento, Italy a.valitutti@email.unitn.it Rafael A. Calvo School of Electrical and Information Engineering University of Sydney Sydney, Australia rafa@ee.usyd.edu.au Abstract In this paper we present an evaluation of new techniques for automatically detecting emo- tions in text. The study estimates categorical model and dimensional model for the recogni- tion of four affective states: Anger, Fear, Joy, and Sadness that are common emotions in three datasets: SemEval-2007 “Affective Text”, ISEAR (International Survey on Emo- tion Antecedents and Reactions), and child- ren’s fairy tales. In the first model, WordNet- Affect is used as a linguistic lexical resource and three dimensionality reduction techniques are evaluated: Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (PLSA), and Non-negative Matrix Factoriza- tion (NMF). In the second model, ANEW (Af- fective Norm for English Words), a normative database with affective terms, is employed. Experiments show that a categorical model us- ing NMF results in better performances for SemEval and fairy tales, whereas a dimension- al model performs better with ISEAR. 1 Introduction Supervised and unsupervised approaches have been used to automatically recognize expressions of emotion in text such as happiness, sadness, anger, etc… Supervised learning techniques have the disadvantage that large annotated data- sets are required for training. Since the emotional interpretations of a text can be highly subjective, more than one annotator is needed, and this makes the process of the annotation very time consuming and expensive. For this reason, unsu- pervised methods are normally preferred in the realm of Natural Language Processing (NLP) and emotions. Supervised and unsupervised techniques have been compared before. (Strapparava and Mihal- cea 2008) describe the comparison between a supervised (Naïve Bayes) and an unsupervised (Latent Semantic Analysis - LSA) method for recognizing six basic emotions. These techniques have been applied to many areas, particularly in improving Intelligent Tutor- ing Systems. For example, (D’Mello, Craig et al. 2008) used LSA but for detecting utterance types and affect in students’ dialogue within Autotutor. (D'Mello, Graesser et al. 2007) proposed five categories for describing the affect states in stu- dent-system dialogue. Significant differences arise not only between these two types of techniques but also between different emotion models, and these differences have significant implications in all these areas. While considering emotions and learning, (Kort, Reilly et al. 2001) proposed (but provided no empirical evidence) a model that combines two emotion models, placing categories in a valence- arousal plane. This mixed approach has also been used in other domains such as blog posts where (Aman and Szpakowicz 2007) studied how to identify emotion categories as well as emotion intensity. To date, many researchers have, how- ever, utilized and evaluated supervised …

Content truncated to 500 words