SteM at SemEval-2016 Task 4: Applying Active Learning to Improve Sentiment Classification

Proceedings of SemEval-2016, pages 64–70, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SteM at SemEval-2016 Task 4: Applying Active Learning to Improve Sentiment Classification Stefan R¨abigera, Mishal Kazmia, Y¨ucel Saygına, Peter Sch¨ullerb, Myra Spiliopoulouc aSabanci University, Istanbul, Turkey bMarmara University, Istanbul, Turkey cOtto-von-Guericke University, Magdeburg, Germany {stefan,mishalkazmi,ysaygin}@sabanciuniv.edu peter.schuller@marmara.edu.tr myra@iti.cs.uni-magdeburg.de Abstract This paper describes our approach to the Se- mEval 2016 task 4, “Sentiment Analysis in Twitter”, where we participated in subtask A. Our system relies on AlchemyAPI and Senti- WordNet to create 43 features based on which we select a feature subset as final represen- tation. Active Learning then filters out noisy tweets from the provided training set, leaving a smaller set of only 900 tweets which we use for training a Multinomial Naive Bayes classi- fier to predict the labels of the test set with an F1 score of 0.478. 1 Introduction Gaining an overview of opinions on recent events or trends is an appealing feature of Twitter. For ex- ample, receiving real-time feedback from the pub- lic about a politician’s speech provides insights to media for the latest polls, analysts and interested in- dividuals including the politician herself. However, detecting tweet sentiment still poses a challenge due to the frequent use of informal language, acronyms, neologisms which constantly change, and the short- ness of tweets, which are limited to 140 characters. SemEval’s subtask 4A (Nakov et al., 2016) deals with the sentiment classification of single tweets into one of the classes “positive”, “neutral” or “nega- tive”. Concretely a training set of 5481 tweets and a development set comprising 1799 tweets was given, and the sentiment of 32009 tweets in a test set had to be predicted and was evaluated using F1-score. The distribution of labels for the given datasets is depicted in Table 1. Tweets with positive polarity outnumber the other two classes and the least num- ber of instances is available for negative tweets. Ini- tially, the organizers provided 6000 tweets for the training set and 2000 tweets for the development set, but only the tweet IDs were released to abide by the Twitter terms of agreement. By the time we down- loaded the data, around 10% of the tweets (519 in the training set, 201 in the test set) were not avail- able anymore. For further details about the labeling process and the datasets see (Nakov et al., 2016). Lexicon-based approaches have done very well in this competition over the past years, i.e, the winners of the years 2013-2015 (Mohammad et al., 2013; Miura et al., 2014; Hagen et al., 2015) relied heav- ily on them. Our goal is to explore alternatives and to complement lexicon-based strategies. The SteM system performs preprocessing, including canoni- calization of tweets, and based on that we extract 43 features as our representation, some features are based on AlchemyAPI1 and SentiWordNet (Esuli and Sebastiani, 2006). We choose 28 of the fea- tures as final representation and learn three classi- fiers based on different subsets of these 28 …

Content truncated to 500 words