SemEval-2013 Task 2: Sentiment Analysis in Twitter

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 312–320, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 2: Sentiment Analysis in Twitter Preslav Nakov QCRI, Qatar Foundation pnakov@qf.org.qa Zornitsa Kozareva USC Information Sciences Institute kozareva@isi.edu Alan Ritter University of Washington aritter@cs.washington.edu Sara Rosenthal Columbia University sara@cs.columbia.edu Veselin Stoyanov JHU HLTCOE ves@cs.jhu.edu Theresa Wilson JHU HLTCOE taw@jhu.edu Abstract In recent years, sentiment analysis in social media has attracted a lot of research interest and has been used for a number of applica- tions. Unfortunately, research has been hin- dered by the lack of suitable datasets, com- plicating the comparison between approaches. To address this issue, we have proposed SemEval-2013 Task 2: Sentiment Analysis in Twitter, which included two subtasks: A, an expression-level subtask, and B, a message- level subtask. We used crowdsourcing on Amazon Mechanical Turk to label a large Twitter training dataset along with additional test sets of Twitter and SMS messages for both subtasks. All datasets used in the evaluation are released to the research community. The task attracted significant interest and a total of 149 submissions from 44 teams. The best- performing team achieved an F1 of 88.9% and 69% for subtasks A and B, respectively. 1 Introduction In the past decade, new forms of communication, such as microblogging and text messaging have emerged and become ubiquitous. Twitter messages (tweets) and cell phone messages (SMS) are often used to share opinions and sentiments about the sur- rounding world, and the availability of social con- tent generated on sites such as Twitter creates new opportunities to automatically study public opinion. Working with these informal text genres presents new challenges for natural language processing be- yond those encountered when working with more traditional text genres such as newswire. Tweets and SMS messages are short in length: a sentence or a headline rather than a document. The language they use is very informal, with creative spelling and punctuation, misspellings, slang, new words, URLs, and genre-specific terminology and abbreviations, e.g., RT for re-tweet and #hashtags.1 How to handle such challenges so as to automati- cally mine and understand the opinions and senti- ments that people are communicating has only very recently been the subject of research (Jansen et al., 2009; Barbosa and Feng, 2010; Bifet et al., 2011; Davidov et al., 2010; O’Connor et al., 2010; Pak and Paroubek, 2010; Tumasjan et al., 2010; Kouloumpis et al., 2011). Another aspect of social media data, such as Twit- ter messages, is that they include rich structured in- formation about the individuals involved in the com- munication. For example, Twitter maintains infor- mation about who follows whom. Re-tweets (re- shares of a tweet) and tags inside of tweets provide discourse information. Modeling such structured in- formation is important because it provides means for empirically studying social interactions where opin- ion is conveyed, e.g., we can study the properties of persuasive language or those associated with influ- ential users. …

Content truncated to 500 words