SentiKLUE: Updating a Polarity Classifier in 48 Hours

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 551–555, Dublin, Ireland, August 23-24, 2014. SentiKLUE: Updating a Polarity Classifier in 48 Hours Stefan Evert and Thomas Proisl and Paul Greiner and Besim Kabashi Friedrich-Alexander-Universität Erlangen-Nürnberg Department Germanistik und Komparatistik Professur für Korpuslinguistik Bismarckstr. 6, 91054 Erlangen, Germany {stefan.evert,thomas.proisl,paul.greiner,besim.kabashi}@fau.de Abstract SentiKLUE is an update of the KLUE po- larity classifier – which achieved good and robust results in SemEval-2013 with a sim- ple feature set – implemented in 48 hours. 1 Introduction The SemEval-2014 shared task on “Sentiment Analysis in Twitter” (Rosenthal et al., 2014) is a re- run of the corresponding shared task from SemEval- 2013 (Nakov et al., 2013) with new test data. It focuses on polarity classification in computer- mediated communication such as Twitter, other micro-blogging services, and SMS. There are two subtasks: the goal of Message Polarity Classifica- tion (B) is to classify an entire SMS, tweet or other message as positive (pos), negative (neg) or neutral (ntr); in the subtask on Contextual Polarity Disam- biguation (A), a single word or short phrase has to be classified in the context of the whole message. The training data are the same as in SemEval- 2013. The test data from 2013 are used as a devel- opment set in order to select features and tune ma- chine learning algorithms, but may not be included in the training data. The 2014 test set comprises the development data, new Twitter messages, Live- Journal entries as out-of-domain data, and a small number of tweets containing sarcasm (see Rosen- thal et al. (2014) for further details). For subtask B, there are 10,239 training items, 5,907 items in the development set, and 3,861 additional unseen items in the new test set. For subtask A, there are 9,505 training items, 6,769 items in the development set, and 3,912 additional items in the test set. Our team participated in the SemEval-2013 shared task with a relatively simple, but robust This work is licensed under a Creative Commons Attribution 4.0 International License. Page numbers and proceedings footer are added by the organisers. Licence details: http: //creativecommons.org/licenses/by/4.0/ system (KLUE) based on a maximum entropy clas- sifier and a small set of features (Proisl et al., 2013). Despite its simplicity, KLUE performed very well in subtask B, ranking 5th out of 36 constrained systems on the Twitter data and 3rd out of 28 on the SMS data. Results for contextual polarity dis- ambiguation (subtask A) were less encouraging, with rank 14 out of 21 constrained systems on the Twitter data and rank 12 out of 19 on the SMS data. This paper describes our efforts to bring the KLUE system up to date within a period of 48 hours. The results obtained by the new SentiKLUE system are summarised in Table 1, showing that the update was successful. The ranking of the system has improved substantially in subtask A, making it one of the best-performing systems in the shared task. Rankings in subtask B are …

Content truncated to 500 words