NRU-HSE at SemEval-2017 Task 4: Tweet Quantification Using Deep Learning Architecture

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 683–688, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics NRU-HSE at SemEval-2017 Task 4: Tweet Quantification Using Deep Learning Architecture Nikolay Karpov National Research University Higher School of Economics 25/12 Bolshaja Pecherskaja str. 603155 Nizhny Novgorod, Russia nkarpov@hse.ru Abstract In many areas, such as social science, politics or market research, people need to deal with dataset shifting over time. Dis- tribution drift phenomenon usually ap- pears in the field of sentiment analysis, when proportions of instances are chang- ing over time. In this case, the task is to correctly estimate proportions of each sen- timent expressed in the set of documents (quantification task). Basically, our study was aimed to analyze the effectiveness of a mixture of quantification technique with one of deep learning architecture. All the techniques are evaluated using the SemEval-2017 Task4 dataset and source code, mentioned in this paper and availa- ble online in the Python programming language. The results of an application of the quantification techniques are dis- cussed. 1 Introduction A traditional classification task is often based on the assumption that data for training a classifier represent test data. But in many areas, such as customer-relationship management or opinion mining, people need to deal with dataset shift or population drift phenomenon. The simplest type of dataset shift is when training set and test set vary only in the distribution of the classes of the instances aka distribution drift. If we would like to measure this variation, the task of accurate classification of each item is replaced by the task of providing accurate proportions of instances from each class (quantification). George Forman suggested defining the ‘quantification task’ as finding the best estimate for the amount of cases in each class in a test set, using a training set with a substantially different class distribution (For- man, 2008). Application of the quantification approach in opinion mining (Esuli et al., 2010), network- behavior analysis (Tang et al., 2010), word-sense disambiguation (Chan and Ng, 2006), remote sensing (Guerrero-Curieses et al., 2009), quality control (Sánchez et al., 2008), monitoring support- call logs (Forman et al., 2006) and credit scoring (Hand and others, 2006) showed high perfor- mance even with a relatively small training set. Although quantification techniques are able to provide accurate sentiment analysis of proportions in situations of distribution drift, the question of an optimal technique for analysis of tweets still raises a lot of questions. It is worth mentioning that sentiment analysis of tweets presents addi- tional challenges to natural language processing, because of the small amount of text (less than 140 characters in each document), usage of creative spelling (e.g. “happpyyy”, “some1 yg bner2 tulus”), abbreviations (such as “wth” or “lol”), in- formal constructions (“hahahaha yava quiet so !ma I m bored av even home nw”) and hashtags (BREAKING: US GDP growth is back! #kid- ding), which are a type of tagging for Twitter mes- sages. We participated in D and E subtasks …

Content truncated to 500 words