JOINT_FORCES: Unite Competing Sentiment Classifiers with Random Forest

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 366–369, Dublin, Ireland, August 23-24, 2014. JOINT_FORCES: Unite Competing Sentiment Classifiers with Random Forest Oliver Dürr, Fatih Uzdilli, and Mark Cieliebak Zurich University of Applied Sciences Winterthur, Switzerland {dueo, uzdi, ciel}@zhaw.ch Abstract In this paper, we describe how we cre- ated a meta-classifier to detect the mes- sage-level sentiment of tweets. We par- ticipated in SemEval-2014 Task 9B by combining the results of several exist- ing classifiers using a random forest. The results of 5 other teams from the competition as well as from 7 general- purpose commercial classifiers were used to train the algorithm. This way, we were able to get a boost of up to 3.24 F1 score points. 1 Introduction The interest in sentiment analysis grows as pub- licly available text content grows. As one of the most used social media platforms, Twitter pro- vides its users a unique way of expressing them- selves. Thus, sentiment analysis of tweets has become a hot research topic among academia and industry. In this paper, we describe our approach of com- bining multiple sentiment classifiers into a meta- classifier. The introduced system participated in SemEval-2014 Task 9: “Sentiment Analysis in Twitter, Subtask–B Message Polarity Classifica- tion” (Rosenthal et al., 2014). The goal was to classify a tweet on the message level using the three classes positive, negative, and neutral. The performance is measured using the macro- averaged F1 score of the positive and negative classes which is simply named “F1 score” throughout the paper. An almost identical task was already run in 2013 (Nakov et al., 2013). The tweets for training and development were only provided as tweet ids. A fraction (10-15%) of the tweets was no longer available on twitter, which makes the results of the competition not fully comparable. For testing, in addition to last year’s data (tweets and SMS) new tweets and data from a surprise domain (LiveJournal) were provided. An overview of the provided data is shown in Table 1. Using additional manually labelled data for training the algorithm was not allowed for a “constrained” submission. Submissions using additional data for training were marked as “un- constrained”. Dataset Total Pos Neg Neu Training (Tweets) 8224 3058 1210 3956 Dev (Tweets) 1417 494 286 637 Test: Twitter2013 3813 1572 601 1640 Test: SMS2013 2093 492 394 1207 Test: Twitter2014 1853 982 202 669 Test: Twitter’14Sarcasm 86 33 40 13 Test: LiveJournal2014 1142 427 304 411 Table 1: Number of Documents we were able to download for Training, Development and Test- ing. Our System. The results of 5 other teams from the competition as well as from 7 general- purpose commercial classifiers were used to train our algorithm. Scientific subsystems were s_gez (Gezici et al., 2013), s_jag (Jaggi et al., 2014), s_mar (Marchand et al., 2013), s_fil (Filho and Pardo, 2013), s_gun (Günther and Furrer, 2013). They are all “constrained” and machine learning- based, some with hybrid rule-based approaches. Commercial subsystems were provided by This work …

Content truncated to 500 words