SemEval-2016 Task 4: Sentiment Analysis in Twitter

Proceedings of SemEval-2016, pages 1–18, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 4: Sentiment Analysis in Twitter Preslav Nakov♣, Alan Ritter♦, Sara Rosenthal♥, Fabrizio Sebastiani♣∗, Veselin Stoyanov♠ ♣Qatar Computing Research Institute, Hamad bin Khalifa University, Qatar ♦Department of Computer Science and Engineering, The Ohio State University, USA ♥IBM Watson Health Research, USA ♠Johns Hopkins University, USA Abstract This paper discusses the fourth year of the ”Sentiment Analysis in Twitter Task”. SemEval-2016 Task 4 comprises five sub- tasks, three of which represent a significant departure from previous editions. The first two subtasks are reruns from prior years and ask to predict the overall sentiment, and the sentiment towards a topic in a tweet. The three new subtasks focus on two variants of the basic “sentiment classification in Twitter” task. The first variant adopts a five-point scale, which confers an ordinal character to the clas- sification task. The second variant focuses on the correct estimation of the prevalence of each class of interest, a task which has been called quantification in the supervised learn- ing literature. The task continues to be very popular, attracting a total of 43 teams. 1 Introduction Sentiment classification is the task of detecting whether a textual item (e.g., a product review, a blog post, an editorial, etc.) expresses a POSI- TIVE or a NEGATIVE opinion in general or about a given entity, e.g., a product, a person, a political party, or a policy. Sentiment classification has be- come a ubiquitous enabling technology in the Twit- tersphere. Classifying tweets according to sentiment has many applications in political science, social sci- ences, market research, and many others (Mart´ınez- C´amara et al., 2014; Mejova et al., 2015). ∗Fabrizio Sebastiani is currently on leave from Consiglio Nazionale delle Ricerche, Italy. As a testament to the prominence of research on sentiment analysis in Twitter, the tweet sentiment classification (TSC) task has attracted the highest number of participants in the last three SemEval campaigns (Nakov et al., 2013; Rosenthal et al., 2014; Rosenthal et al., 2015; Nakov et al., 2016b). Previous editions of the SemEval task involved binary (POSITIVE vs. NEGATIVE) or single-label multi-class classification (SLMC) when a NEU- TRAL1 class is added (POSITIVE vs. NEGATIVE vs. NEUTRAL). SemEval-2016 Task 4 represents a sig- nificant departure from these previous editions. Al- though two of the subtasks (Subtasks A and B) are reincarnations of previous editions (SLMC classifi- cation for Subtask A, binary classification for Sub- task B), SemEval-2016 Task 4 introduces two com- pletely new problems, taken individually (Subtasks C and D) and in combination (Subtask E): 1.1 Ordinal Classification We replace the two- or three-point scale with a five- point scale {HIGHLYPOSITIVE, POSITIVE, NEU- TRAL, NEGATIVE, HIGHLYNEGATIVE}, which is now ubiquitous in the corporate world where hu- man ratings are involved: e.g., Amazon, TripAdvi- sor, and Yelp, all use a five-point scale for rating sen- timent towards products, hotels, and restaurants. Moving from a categorical two/three-point scale to an ordered five-point scale means, in machine learning …

Content truncated to 500 words