Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon

Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 26–34, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon Saif M. Mohammad and Peter D. Turney Institute for Information Technology, National Research Council Canada. Ottawa, Ontario, Canada, K1A 0R6 {saif.mohammad,peter.turney}@nrc-cnrc.gc.ca Abstract Even though considerable attention has been given to semantic orientation of words and the creation of large polarity lexicons, research in emotion analysis has had to rely on lim- ited and small emotion lexicons. In this pa- per, we show how we create a high-quality, moderate-sized emotion lexicon using Me- chanical Turk. In addition to questions about emotions evoked by terms, we show how the inclusion of a word choice question can dis- courage malicious data entry, help identify in- stances where the annotator may not be famil- iar with the target term (allowing us to reject such annotations), and help obtain annotations at sense level (rather than at word level). We perform an extensive analysis of the annota- tions to better understand the distribution of emotions evoked by terms of different parts of speech. We identify which emotions tend to be evoked simultaneously by the same term and show that certain emotions indeed go hand in hand. 1 Introduction When analyzing text, automatically detecting emo- tions such as joy, sadness, fear, anger, and surprise is useful for a number of purposes, including identify- ing blogs that express specific emotions towards the topic of interest, identifying what emotion a news- paper headline is trying to evoke, and devising auto- matic dialogue systems that respond appropriately to different emotional states of the user. Often different emotions are expressed through different words. For example, delightful and yummy indicate the emo- tion of joy, gloomy and cry are indicative of sadness, shout and boiling are indicative of anger, and so on. Therefore an emotion lexicon—a list of emotions and words that are indicative of each emotion—is likely to be useful in identifying emotions in text. Words may evoke different emotions in different contexts, and the emotion evoked by a phrase or a sentence is not simply the sum of emotions conveyed by the words in it, but the emotion lexicon will be a useful component for any sophisticated emotion de- tecting algorithm. The lexicon will also be useful for evaluating automatic methods that identify the emo- tions evoked by a word. Such algorithms may then be used to automatically generate emotion lexicons in languages where no such lexicons exist. As of now, high-quality high-coverage emotion lexicons do not exist for any language, although there are a few limited-coverage lexicons for a handful of lan- guages, for example, the WordNet Affect Lexicon (WAL) (Strapparava and Valitutti, 2004) for six ba- sic emotions and the General Inquirer (GI) (Stone et al., 1966), which categorizes words into a number of categories, including positive and negative semantic orientation. Amazon has …

Content truncated to 500 words