NRU-HSE at SemEval-2016 Task 4: Comparative Analysis of Two Iterative Methods Using Quantification Library

Proceedings of SemEval-2016, pages 171–177, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics NRU-HSE at SemEval-2016 Task 4: Comparative Analysis of Two Iterative Methods Using Quantification Library Nikolay Karpov, Alexander Porshnev, Kirill Rudakov National Research University Higher School of Economics 25/12 Bolshaja Pecherskaja str. 603155 Nizhny Novgorod, Russia {nkarpov, aporshnev}@hse.ru, rudakovkirillx@gmail.com Abstract In many areas, such as social science, politics or market research, people need to track sentiment and their changes over time. For sentiment analysis in this field it is more important to correctly esti- mate proportions of each sentiment expressed in the set of documents (quantification task) than to accurately estimate sentiment of a particular doc- ument (classification). Basically, our study was aimed to analyze the effectiveness of two iterative quantification techniques and to compare their ef- fectiveness with baseline methods. All the tech- niques are evaluated using a set of synthesized data and the SemEval-2016 Task4 dataset. We made the quantification methods from this paper available as a Python open source library. The results of com- parison and possible limitations of the quantifica- tion techniques are discussed. 1 Introduction In many areas, such as customer-relationship manage- ment or opinion mining, people need to track changes over time and measure proportions of documents ex- pressing different sentiments. In these situations, the task of accurate categorization of each document is re- placed by the task of providing accurate proportions of documents from each class (quantification). George Forman suggested defining the ‘quantification task’ as finding the best estimate for the amount of cases in each class in a test set, using a training set with substantially different class distribution (Forman, 2008). Application of the quantification approach in opinion mining (Esuli et al., 2010), network-behavior analysis (Tang et al., 2010), word-sense disambiguation (Chan and Ng, 2006), remote sensing (Guerrero-Curieses et al., 2009), quality control (Sánchez et al., 2008), moni- toring support-call logs (Forman et al., 2006) and credit scoring (Hand and others, 2006) showed high perfor- mance even with a relatively small training set. Although quantification techniques are able to pro- vide accurate sentiment analysis of proportions in situa- tions of distribution drift, the question of optimal tech- nique for analysis of tweets still raises a lot of questions. It is worth mentioning that sentiment analysis of tweets presents additional challenges to natural language pro- cessing, because of the small amount of text (less than 140 characters in each document), usage of creative spelling (e.g. “happpyyy”, “some1 yg bner2 tulus”), ab- breviations (such as “wth” or “lol”), informal construc- tions (“hahahaha yava quiet so !ma I m bored av even home nw”) and hashtags (BREAKING: US GDP growth is back! #kidding), which are a type of tagging for Twitter messages. In our paper we used several quantification methods mentioned in literature as the best ones and evaluated them by comparing their effectiveness with one another and with baseline methods. The paper is organized as follows. In Section 2, we first look at the notation, then we briefly overview …

Content truncated to 500 words