SemEval-2015 Task 10: Sentiment Analysis in Twitter

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 451–463, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 10: Sentiment Analysis in Twitter Sara Rosenthal Columbia University sara@cs.columbia.edu Saif M Mohammad National Research Council Canada saif.mohammad@nrc-cnrc.gc.ca Preslav Nakov Qatar Computing Research Institute pnakov@qf.org.qa Alan Ritter The Ohio State University aritter@cs.washington.edu Svetlana Kiritchenko National Research Council Canada Svetlana.Kiritchenko@nrc-cnrc.gc.ca Veselin Stoyanov Facebook vesko.st@gmail.com Abstract In this paper, we describe the 2015 iteration of the SemEval shared task on Sentiment Analy- sis in Twitter. This was the most popular sen- timent analysis shared task to date with more than 40 teams participating in each of the last three years. This year’s shared task competi- tion consisted of five sentiment prediction sub- tasks. Two were reruns from previous years: (A) sentiment expressed by a phrase in the context of a tweet, and (B) overall sentiment of a tweet. We further included three new sub- tasks asking to predict (C) the sentiment to- wards a topic in a single tweet, (D) the over- all sentiment towards a topic in a set of tweets, and (E) the degree of prior polarity of a phrase. 1 Introduction Social media such as Weblogs, microblogs, and dis- cussion forums are used daily to express personal thoughts, which allows researchers to gain valuable insight into the opinions of a very large number of individuals, i.e., at a scale that was simply not pos- sible a few years ago. As a result, nowadays, sen- timent analysis is commonly used to study the pub- lic opinion towards persons, objects, and events. In particular, opinion mining and opinion detection are applied to product reviews (Hu and Liu, 2004), for agreement detection (Hillard et al., 2003), and even for sarcasm identification (Gonz´alez-Ib´a˜nez et al., 2011; Liebrecht et al., 2013). Early work on detecting sentiment focused on newswire text (Wiebe et al., 2005; Baccianella et al., 2010; Pang et al., 2002; Hu and Liu, 2004). As later research turned towards social media, people real- ized this presented a number of new challenges. Misspellings, poor grammatical structure, emoti- cons, acronyms, and slang were common in these new media, and were explored by a number of re- searchers (Barbosa and Feng, 2010; Bifet et al., 2011; Davidov et al., 2010; Jansen et al., 2009; Kouloumpis et al., 2011; O’Connor et al., 2010; Pak and Paroubek, 2010). Later, specialized shared tasks emerged, e.g., at SemEval (Nakov et al., 2013; Rosenthal et al., 2014), which compared teams against each other in a controlled environment us- ing the same training and testing datasets. These shared tasks had the side effect to foster the emer- gence of a number of new resources, which eventu- ally spread well beyond SemEval, e.g., NRC’s Hash- tag Sentiment lexicon and the Sentiment140 lexicon (Mohammad et al., 2013).1 Below, we discuss the public evaluation done as part of SemEval-2015 Task 10. In its third year, the SemEval task on Sentiment Analysis in Twitter has once again attracted a large …

Content truncated to 500 words