Choosing the Right Words: Characterizing and Reducing Error of the Word Count Approach

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 296–305, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics Choosing the Right Words: Characterizing and Reducing Error of the Word Count Approach H. Andrew Schwartz,1 Johannes Eichstaedt,1 Lukasz Dziurzynski,1 Eduardo Blanco,2 Margaret L. Kern,1 Stephanie Ramones,1 Martin Seligman,1 and Lyle Ungar1 1University of Pennsylvania 2Lymba Corporation hansens@seas.upenn.edu Abstract Social scientists are increasingly using the vast amount of text available on social me- dia to measure variation in happiness and other psychological states. Such studies count words deemed to be indicators of happiness and track how the word frequencies change across locations or time. This word count ap- proach is simple and scalable, yet often picks up false signals, as words can appear in differ- ent contexts and take on different meanings. We characterize the types of errors that occur using the word count approach, and find lex- ical ambiguity to be the most prevalent. We then show that one can reduce error with a simple refinement to such lexica by automat- ically eliminating highly ambiguous words. The resulting refined lexica improve precision as measured by human judgments of word oc- currences in Facebook posts. 1 Introduction Massive social media corpora, such as blogs, tweets, and Facebook statuses have recently peaked the in- terest of social scientists. Compared to traditional samples in tens or hundreds, social media sample sizes are orders of magnitude larger, often contain- ing millions or billions of posts or queries. Such text provides potential for unobtrusive, inexpensive, and real-time measurement of psychological states (such as positive or negative affect) and aspects of sub- jective well-being (such as happiness and engage- ment). Social scientists have recently begun to use social media text in a variety of studies (Cohn et al., 2004; Kramer, 2010; Tausczik and Pennebaker, 2010; Kamvar and Harris, 2011; Dodds et al., 2011; Golder and Macy, 2011). One of the most popular approaches to estimate psychological states is by using the word count method (Pennebaker et al., 2007), where one tracks the frequency of words that have been judged to be associated with a given state. Greater use of such words is taken to index the prevalence of the cor- responding state. For example, the use of the word ‘happy’ is taken to index positive emotion, and ‘an- gry’ to index negative emotion. The most widely used tool to carry out such analysis, and the one we investigate in this paper, is Pennebaker’s Linguistic Inquiry and Word Count, (LIWC) (Pennebaker et al., 2001; Pennebaker et al., 2007). LIWC, originally de- veloped to analyze writing samples for emotion and control, has grown to include a variety of lexica for linguistic and psychosocial topics including positive and negative emotions, pronouns, money, work, and religion. The word count approach has high appeal to social scientists in need of a tool to approach so- cial media, and although others have been used (see, for example (Gottschalk and …

Content truncated to 500 words