Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling

Proceedings of NAACL-HLT 2016, pages 811–817, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling Svetlana Kiritchenko and Saif M. Mohammad National Research Council Canada {svetlana.kiritchenko,saif.mohammad}@nrc-cnrc.gc.ca Abstract Access to word–sentiment associations is use- ful for many applications, including senti- ment analysis, stance detection, and linguistic analysis. However, manually assigning fine- grained sentiment association scores to words has many challenges with respect to keeping annotations consistent. We apply the annota- tion technique of Best–Worst Scaling to ob- tain real-valued sentiment association scores for words and phrases in three different do- mains: general English, English Twitter, and Arabic Twitter. We show that on all three do- mains the ranking of words by sentiment re- mains remarkably consistent even when the annotation process is repeated with a different set of annotators. We also, for the first time, determine the minimum difference in senti- ment association that is perceptible to native speakers of a language. 1 Introduction Word–sentiment associations, commonly captured in sentiment lexicons, are useful in automatic sen- timent prediction (Pontiki et al., 2014; Rosenthal et al., 2014), stance detection (Mohammad et al., 2016a; Mohammad et al., 2016b), literary analysis (Hartner, 2013; Kleres, 2011; Mohammad, 2012), detecting personality traits (Grijalva et al., 2015; Mohammad and Kiritchenko, 2015), and other ap- plications. Manually created sentiment lexicons are especially useful because they tend to be more accu- rate than automatically generated lexicons; they can be used to automatically generate large-scale lexi- cons (Tang et al., 2014; Esuli and Sebastiani, 2006); they can be used to evaluate different methods of automatically creating sentiment lexicons; and they can be used for linguistic analyses such as examin- ing how sentiment is composed in phrases and sen- tences. The sentiment of a phrase can differ significantly from the sentiment of its constituent words. Sen- timent composition is the determining of sentiment of a multi-word linguistic unit, such as a phrase or a sentence, from its constituents. Lexicons that in- clude sentiment associations for phrases as well as for their constituent words are useful in studying sentiment composition. We refer to them as senti- ment composition lexicons (SCLs). We created SCLs for three domains, and all three were used in recent SemEval shared tasks. We refer to the lexicon cre- ated for the English Twitter domain as the SemEval- 2015 English Twitter Sentiment Lexicon; for the gen- eral English domain as the SemEval-2016 General English Sentiment Modifiers Lexicon; and for the Arabic Twitter domain as the SemEval-2016 Ara- bic Twitter Sentiment Lexicon. Note that the English Twitter lexicon was first described in (Kiritchenko et al., 2014), whereas the other two are novel contribu- tions presented in this paper. Most existing manually created sentiment lex- icons tend to provide only lists of positive and negative words with very coarse levels of senti- ment (Stone et al., 1966; Hu and Liu, 2004; Wil- son et al., 2005; Mohammad and Turney, 2013). The coarse-grained distinctions may be less use- ful in …

Content truncated to 500 words