NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 321–327, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu National Research Council Canada Ottawa, Ontario, Canada K1A 0R6 {saif.mohammad,svetlana.kiritchenko,xiaodan.zhu}@nrc-cnrc.gc.ca Abstract In this paper, we describe how we created two state-of-the-art SVM classifiers, one to de- tect the sentiment of messages such as tweets and SMS (message-level task) and one to de- tect the sentiment of a term within a message (term-level task). Among submissions from 44 teams in a competition, our submissions stood first in both tasks on tweets, obtaining an F-score of 69.02 in the message-level task and 88.93 in the term-level task. We imple- mented a variety of surface-form, semantic, and sentiment features. We also generated two large word–sentiment association lexi- cons, one from tweets with sentiment-word hashtags, and one from tweets with emoticons. In the message-level task, the lexicon-based features provided a gain of 5 F-score points over all others. Both of our systems can be replicated using freely available resources.1 1 Introduction Hundreds of millions of people around the world ac- tively use microblogging websites such as Twitter. Thus there is tremendous interest in sentiment anal- ysis of tweets across a variety of domains such as commerce (Jansen et al., 2009), health (Chew and Eysenbach, 2010; Salath´e and Khandelwal, 2011), and disaster management (Verma et al., 2011; Man- del et al., 2012). 1The three authors contributed equally to this paper. Svet- lana Kiritchenko developed the system for the message-level task, Xiaodan Zhu developed the system for the term-level task, and Saif Mohammad led the overall effort, co-ordinated both tasks, and contributed to feature development. In this paper, we describe how we created two state-of-the-art SVM classifiers, one to detect the sentiment of messages such as tweets and SMS (message-level task) and one to detect the sentiment of a term within a message (term-level task). The sentiment can be one out of three possibilities: posi- tive, negative, or neutral. We developed these classi- fiers to participate in an international competition or- ganized by the Conference on Semantic Evaluation Exercises (SemEval-2013) (Wilson et al., 2013).2 The organizers created and shared sentiment-labeled tweets for training, development, and testing. The distributions of the labels in the different datasets is shown in Table 1. The competition, officially re- ferred to as Task 2: Sentiment Analysis in Twitter, had 44 teams (34 for the message-level task and 23 for the term-level task). Our submissions stood first in both tasks, obtaining a macro-averaged F-score of 69.02 in the message-level task and 88.93 in the term-level task. The task organizers also provided a second test dataset, composed of Short Message Service (SMS) messages (no training data of SMS messages was provided). We applied our classifiers on the SMS test set without any further tuning. Nonetheless, the classifiers still obtained the first position in identify- ing …

Content truncated to 500 words