Amobee at SemEval-2017 Task 4: Deep Learning System for Sentiment Detection on Twitter

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 653–658, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics Amobee at SemEval-2017 Task 4: Deep Learning System for Sentiment Detection on Twitter Alon Rozental∗, Daniel Fleischer∗ Amobee, Tel Aviv, Israel {arozental,danielf}@amobee.com Abstract This paper describes the Amobee senti- ment analysis system, adapted to compete in SemEval 2017 task 4. The system con- sists of two parts: a supervised training of RNN models based on a Twitter sentiment treebank, and the use of feedforward NN, Naive Bayes and logistic regression classi- fiers to produce predictions for the differ- ent sub-tasks. The algorithm reached the 3rd place on the 5-label classification task (sub-task C). 1 Introduction Sentiment detection is the process of determin- ing whether a text has a positive or negative at- titude toward a given entity (topic) or in general. Detecting sentiment on Twitter—a social network where users interact via short 140-character mes- sages, exchanging information and opinions—is becoming ubiquitous. Sentiment in Twitter mes- sages (tweets) can capture the popularity level of political figures, ideas, brands, products and peo- ple. Tweets and other social media texts are chal- lenging to analyze as they are inherently different; use of slang, mis-spelling, sarcasm, emojis and co- mentioning of other messages pose unique diffi- culties. Combined with the vast amount of Twitter data (mostly public), these make sentiment detec- tion on Twitter a focal point for data science re- search. SemEval is a yearly event in which teams com- pete in natural language processing tasks. Task 4 is concerned with sentiment analysis in Twitter; it contains five sub-tasks which include classifica- tion of tweets according to 2, 3 or 5 labels and quantification of sentiment distribution regarding ∗These authors contributed equally to this work. topics mentioned in tweets; for a complete de- scription of task 4 see Rosenthal et al. (2017). This paper describes our system and participa- tion in all sub-tasks of SemEval 2017 task 4. Our system consists of two parts: a recurrent neural network trained on a private Twitter dataset, fol- lowed by a task-specific combination of model stacking and logistic regression classifiers. The paper is organized as follows: section 2 describes the training of RNN models, data be- ing used and model selection; section 3 describes the extraction of semantic features; section 4 de- scribes the task-specific workflows and scores. We review and summarize in section 5. Finally, sec- tion 6 describes our future plans, mainly the de- velopment of an LSTM algorithm. 2 RNN Models The first part of our system consisted of training recursive-neural-tensor-network (RNTN) models (Socher et al., 2013). 2.1 Data Our training data for this part was created by tak- ing a random sample1 from Twitter and having it manually annotated on a 5-label basis to produce fully sentiment-labeled parse-trees, much like the Stanford sentiment treebank. The sample contains twenty thousand tweets with sentiment distribu- tion as following: v-neg. neg. neu. pos. v-pos. Train 8.4% 23.2% 31.7% 25.3% 11.4% Test …

Content truncated to 500 words