IIP at SemEval-2016 Task 4: Prioritizing Classes in Ensemble Classification for Sentiment Analysis of Tweets

Proceedings of SemEval-2016, pages 225–229, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics IIP at SemEval-2016 Task 4: Prioritizing Classes in Ensemble Classification for Sentiment Analysis of Tweets Jasper Friedrichs Infosys Limited 7707 Gateway Boulevard Newark, CA 94560, USA jasper friedrichs@infosys.com Abstract This paper describes the submission of team IIP in SemEval-2016 Task 4 Subtask A. The presented system is a novel weighted sum ensemble approach for sentiment analysis of short informal texts. The ensemble com- bines member classifiers that output classifi- cation confidence metrics. For the ensemble classification decision the members are com- bined by weights. In the presented approach the weights are derived to prioritize specific classes in multi-class classification. The pre- sented results confirm that this improves re- sults for the prioritized classes. The official task submission achieved a macro-averaged negative positive F1 of 57.4%. Post submis- sion changes resulted in a F1 score of 60.2%. The evaluation also shows that the system out- performs other ensemble methods. 1 Introduction The SemEval workshops offer the opportunity to compete across a variety of natural language pro- cessing tasks. The SemEval-2016 Task 4 Sub- task A targets message polarity classification of tweets (Nakov et al., 2016). The polarity can be neg- ative, neutral or positive while the submissions are ranked omitting performance on the neutral class. In practical use cases some classes of a multi- classification problem might be deemed more im- portant than others. For example some work looks explicitly at negative sentiment (Tetlock, 2007). Combining diverse methods has shown success in sentiment analysis. The combination of machine learners and opinion lexicons has resulted in some of the best submissions in previous SemEval competi- tions (Kiritchenko et al., 2014; Miura et al., 2014). Along the line of combining different methods, en- semble approaches have also shown top results in previous runs of this task. Both ensembles of a small number of sophisticated systems (Hagen et al., 2015) as well as large numbers of simpler ap- proaches have been evaluated (Wicentowski, 2015). Ensemble classification with regard to combining different machine learners and feature spaces has previously been evaluated extensively for document level sentiment classification (Xia et al., 2011). In that context, weighted sum ensemble methods have shown the best performance. This paper describes a weighted sum ensemble that prioritizes some classes in multi-class classi- fication. Results compare the system against two baselines. One baseline is the equivalent approach without prioritizing classes, while the other is an un- weighted combination of ensemble members. Naive Bayes and logistic regression classifiers are explored as members across a variety of feature spaces. These classifiers are know to perform differently (Ng and Jordan, 2002). The presented results show: 1. The presented approach successfully prioritizes classes in a multi-class classification problem. 2. The ensemble method outperforms individual members and the baseline ensembles. The system description will start by a brief out- line of the evaluation data. Then the ensemble mem- bers are described before the ensemble method is de- tailed. Finally, the …

Content truncated to 500 words