Identifying Emotions, Intentions, and Attitudes in Text Using a Game with a Purpose

Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 71–79, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Identifying Emotions, Intentions, and Attitudes in Text Using a Game with a Purpose Lisa Pearl Department of Cognitive Sciences University of California, Irvine 3151 Social Science Plaza Irvine, CA 92697, USA lpearl@uci.edu Mark Steyvers Department of Cognitive Sciences University of California, Irvine 3151 Social Science Plaza Irvine, CA 92697, USA msteyver@uci.edu Abstract Subtle social information is available in text such as a speaker’s emotional state, intentions, and attitude, but current information extrac- tion systems are unable to extract this infor- mation at the level that humans can. We de- scribe a methodology for creating databases of messages annotated with social information based on interactive games between humans trying to generate and interpret messages for a number of different social information types. We then present some classification results achieved by using a small-scale database cre- ated with this methodology. 1 Introduction A focus of much information extraction research has been identifying surface-level semantic content (e.g., identifying who did what to whom when). In recent years, research on sentiment analysis and opinion mining has recognized that more subtle in- formation can be communicated via linguistic fea- tures in the text (see Pang and Lee (2008) for a re- view), such as whether text (e.g., a movie review) is positive or negative (Turney 2002, Pang, Lee, and Vaithyanathan 2002, Dave, Lawrence, and Pen- nock 2003, Wiebe et al. 2004, Kennedy and Inkpen 2006, Agarwal, Biadsy, and Mckeown 2009, Greene and Resnik 2009, among many others). However, other subtle information available in text, such as a speaker’s emotional states (e.g., anger, embarrass- ment), intentions (e.g., persuasion, deception), and attitudes (e.g., disbelief, confidence), has not been explored as much, though there has been some work in detecting emotion (e.g., Subasic and Huettner 2001, Alm, Roth, and Sproat 2005, Nicolov et al. 2006, Abbasi 2007) and detecting deception (e.g., Annolli, Balconi, and Ciceri 2002, Zhou et al. 2004, Gupta and Skillicorn 2006, Zhou and Sung 2008). This latter kind of social information is useful for identifying the “tone” of a message, i.e., for un- derstanding the underlying intention behind a mes- sage’s creation, and also for predicting how this message will be interpreted by humans reading it. A technical barrier to extracting this kind of social information is that there are currently no large-scale text databases that are annotated with social infor- mation from which to learn the relevant linguistic cues. That is, there are few examples of social in- formation “ground truth” - text annotated with hu- man perceptions of the social information contained within the text. Given the success of sentiment anal- ysis, we believe this social information could also be retrievable once the relevant linguistic cues are identified. One way to create the necessary annotated data is to draw from computational social science (Lazer et al. 2009), and make use of human-based com- …

Content truncated to 500 words