REACTION: A naive machine learning approach for sentiment classification

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 490–494, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics REACTION: A naive machine learning approach for sentiment classification Silvio Moreira IST/INESC-ID Rua Alves Redol, 9 1000-029 Lisboa Portugal samir@inesc-id.pt Jo˜ao Filgueiras INESC-ID Rua Alves Redol, 9 1000-029 Lisboa Portugal jfilgueiras@inesc-id.pt Bruno Martins IST/INESC-ID Rua Alves Redol, 9 1000-029 Lisboa Portugal bruno.g.martins@ist.utl.pt Francisco Couto LASIGE - FCUL Edif´ıcio C6 Piso 3 Campo Grande 1749 - 016 Lisboa Portugal fcouto@di.fc.ul.pt M´ario J. Silva IST/INESC-ID Rua Alves Redol, 9 1000-029 Lisboa Portugal mjs@inesc-id.pt Abstract We evaluate a naive machine learning ap- proach to sentiment classification focused on Twitter in the context of the sentiment analysis task of SemEval-2013. We employ a classifier based on the Random Forests al- gorithm to determine whether a tweet ex- presses overall positive, negative or neu- tral sentiment. The classifier was trained only with the provided dataset and uses as main features word vectors and lexicon word counts. Our average F-score for all three classes on the Twitter evaluation dataset was 51.55%. The average F-score of both positive and negative classes was 45.01%. For the optional SMS evaluation dataset our overall average F-score was 58.82%. The average between positive and negative F- scores was 50.11%. 1 Introduction Sentiment Analysis is a growing research field, es- pecially on web social networks. In this setting, users share very diverse messages such as real- time reactions to news, events and daily experi- ences. The ability to tap on a vast repository of opinions, such as Twitter, where there is great di- versity of topics, has become an important goal for many different applications. However, due to the nature of the text, NLP systems face additional challenges in this context. Shared messages, such as tweets, are very short and users tend to resort to highly informal an noisy speech. Following this trend, the 2013 edition of Se- mEval1 included a sentiment analysis on Twitter task (SemEval-2013 Task 2). Participants were asked to implement a system capable of determin- ing whether a given tweet expresses positive, neg- ative or neutral sentiment. To help in the develop- ment of the system, an annotated training corpus was released. Systems that used only the given corpus for training were considered constrained, while others were considered unconstrained. The submitted prototypes were evaluated in a dataset consisting of around 3700 tweets of several topics. The metric used was the average F-score between the positive and negative classes. Our goal with this participation was to create a baseline system from which we can build upon and perform experiments to compare new approaches with the state-of-the-art. 2 Related Work The last decade saw a growing interest in systems to automatically process sentiment in text. Many approaches to detect subjectivity and determine 1Proceedings of the 7th International Workshop on Se- mantic Evaluation (SemEval 2013), in conjunction with the Second Joint Conference on Lexical and Computational Se- mantics (*SEM 2013) …

Content truncated to 500 words