NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 443–447, Dublin, Ireland, August 23-24, 2014. NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets Xiaodan Zhu, Svetlana Kiritchenko, and Saif M. Mohammad National Research Council Canada Ottawa, Ontario, Canada K1A 0R6 {xiaodan.zhu,svetlana.kiritchenko,saif.mohammad}@nrc-cnrc.gc.ca Abstract This paper describes state-of-the-art statis- tical systems for automatic sentiment anal- ysis of tweets. In a Semeval-2014 shared task (Task 9), our submissions obtained highest scores in the term-level sentiment classification subtask on both the 2013 and 2014 tweets test sets. In the message-level sentiment classification task, our submis- sions obtained highest scores on the Live- Journal blog posts test set, sarcastic tweets test set, and the 2013 SMS test set. These systems build on our SemEval-2013 senti- ment analysis systems (Mohammad et al., 2013) which ranked first in both the term- and message-level subtasks in 2013. Key improvements over the 2013 systems are in the handling of negation. We create separate tweet-specific sentiment lexicons for terms in affirmative contexts and in negated contexts. 1 Introduction Automatically detecting sentiment of tweets (and other microblog posts) has attracted extensive interest from both the academia and industry. The Conference on Semantic Evaluation Exercises (SemEval) organizes a shared task on the senti- ment analysis of tweets with two subtasks. In the message-level task, the participating systems are to identify whether a tweet as a whole expresses positive, negative, or neutral sentiment. In the term-level task, the objective is to determine the sentiment of a marked target term (a single word or a multi-word expression) within the tweet. Our submissions stood first in both subtasks in 2013. This paper describes improvements over that sys- Evaluation Set Term-level Task Message-level Task Twt14 1 4 Twt13 1 2 Sarc14 3 1 LvJn14 2 1 SMS13 2 1 Table 1: Overall rank of NRC-Canada sentiment analysis models in Semeval-2014 Task 9 under the constrained condition. The rows are five evalua- tion datasets and the columns are the two subtasks. tem and the subsequent submissions to the 2014 shared task (Rosenthal et al., 2014). The training data for the SemEval-2014 shared task is same as that of SemEval-2013 (about 10,000 tweets). The 2014 test set has five sub- categories: a tweet set provided newly in 2014 (Twt14), the tweet set used for testing in the 2013 shared task (Twt13), a set of tweets that are sarcas- tic (Sarc14), a set of sentences from the blogging website LiveJournal (LvJn14), and the set of SMS messages used for testing in the 2013 shared task (SMS13). Instances from these categories were in- terspersed in the provided test set. The partici- pants were not told about the source of the indi- vidual messages. The objective was to determine how well a system trained on tweets generalizes to texts from other domains. Our submissions to SemEval-2014 Task 9, ranked first in five out of the ten subtask–dataset combinations. In the other evaluation sets as well, our submissions performed competitively. The results are summarized in Table 1. As …

Content truncated to 500 words