SemEval-2014 Task 9: Sentiment Analysis in Twitter

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 9: Sentiment Analysis in Twitter Sara Rosenthal Columbia University sara@cs.columbia.edu Preslav Nakov Qatar Computing Research Institute pnakov@qf.org.qa Alan Ritter Carnegie Mellon University rittera@cs.cmu.edu Veselin Stoyanov Johns Hopkins University ves@cs.jhu.edu Abstract We describe the Sentiment Analysis in Twitter task, ran as part of SemEval-2014. It is a continuation of the last year’s task that ran successfully as part of SemEval- 2013. As in 2013, this was the most popu- lar SemEval task; a total of 46 teams con- tributed 27 submissions for subtask A (21 teams) and 50 submissions for subtask B (44 teams). This year, we introduced three new test sets: (i) regular tweets, (ii) sarcas- tic tweets, and (iii) LiveJournal sentences. We further tested on (iv) 2013 tweets, and (v) 2013 SMS messages. The highest F1- score on (i) was achieved by NRC-Canada at 86.63 for subtask A and by TeamX at 70.96 for subtask B. 1 Introduction In the past decade, new forms of communica- tion have emerged and have become ubiquitous through social media. Microblogs (e.g., Twitter), Weblogs (e.g., LiveJournal) and cell phone mes- sages (SMS) are often used to share opinions and sentiments about the surrounding world, and the availability of social content generated on sites such as Twitter creates new opportunities to au- tomatically study public opinion. Working with these informal text genres presents new challenges for natural language pro- cessing beyond those encountered when work- ing with more traditional text genres such as newswire. The language in social media is very informal, with creative spelling and punctuation, misspellings, slang, new words, URLs, and genre- specific terminology and abbreviations, e.g., RT for re-tweet and #hashtags1. This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ 1Hashtags are a type of tagging for Twitter messages. Moreover, tweets and SMS messages are short: a sentence or a headline rather than a document. How to handle such challenges so as to automat- ically mine and understand people’s opinions and sentiments has only recently been the subject of research (Jansen et al., 2009; Barbosa and Feng, 2010; Bifet et al., 2011; Davidov et al., 2010; O’Connor et al., 2010; Pak and Paroubek, 2010; Tumasjan et al., 2010; Kouloumpis et al., 2011). Several corpora with detailed opinion and sen- timent annotation have been made freely avail- able, e.g., the MPQA newswire corpus (Wiebe et al., 2005), the movie reviews corpus (Pang et al., 2002), or the restaurant and laptop reviews cor- pora that are part of this year’s SemEval Task 4 (Pontiki et al., 2014). These corpora have proved very valuable as resources for learning about the language of sentiment in general, but they do not focus on tweets. While some Twitter sentiment datasets were created prior to SemEval-2013, they were either small and proprietary, such as the i- sieve corpus (Kouloumpis et …

Content truncated to 500 words