RuSentiment: An Enriched Sentiment Analysis Dataset for Social Media in Russian

Proceedings of the 27th International Conference on Computational Linguistics, pages 755–763 Santa Fe, New Mexico, USA, August 20-26, 2018. 755 RuSentiment: An Enriched Sentiment Analysis Dataset for Social Media in Russian Anna Rogers†, Alexey Romanov†, Anna Rumshisky†, Svitlana Volkova‡, Mikhail Gronas§, Alex Gribov† †Dept. of Computer Science, University of Massachusetts Lowell, Lowell, MA, USA {arogers,aromanov,arum}@cs.uml.edu,alexander_gribov@student.uml.edu ‡Pacific Northwest National Laboratory, Richland, WA, USA svitlana.volkova@pnnl.gov §Dept. of Russian, Dartmouth College, Hanover, NH, USA mikhail.gronas@dartmouth.edu Abstract This paper presents RuSentiment, a new dataset for sentiment analysis of social media posts in Russian, and a new set of comprehensive annotation guidelines that are extensible to other lan- guages. RuSentiment is currently the largest in its class for Russian, with 31,185 posts annotated with Fleiss’ kappa of 0.58 (3 annotations per post). To diversify the dataset, 6,950 posts were pre-selected with an active learning-style strategy. We report baseline classification results, and we also release the best-performing word embeddings trained on 3.2B corpus of Russian social media posts. 1 Introduction Over the past several years sentiment analysis has been increasingly important in political science (Ceron et al., 2015) and journalism (Jiang et al., 2017). Such applications necessitate resources for languages spoken in the conflict zones. Our study focuses on Russian, which to date has little annotated data (Loukachevitch and Rubtsova, 2016; Koltsova et al., 2016), and no openly available sentiment detection systems beyond the dictionary-based ones. However, lexical features have time and again shown inferior performance compared to the supervised learning approaches using annotated data (Gombar et al., 2017), and lack of such a resource severely limits sentiment analysis applications for Russian. We present RuSentiment, a dataset of public posts on VKontakte (VK), the largest Russian social network that currently boasts about 100M monthly active users.1 RuSentiment was developed with new comprehensive guidelines that enabled light and speedy annotation while maintaining consistent coverage of a wide range of explicitly and implicitly expressed sentiment. The overall inter-annotator agreement in terms of Fleiss’ kappa stands at 0.58. In total, 31,185 posts were annotated, 21,268 of which were selected randomly (including 2,967 for the test set). 6,950 posts were pre-selected with an active learning-style strategy in order to diversify the data. This makes RuSentiment the largest openly available sentiment dataset for social media, and the largest general domain sentiment dataset for this relatively low-resource language. We also present baseline classification results on the new dataset. The best results were achieved with a neural network model that made use of word embeddings trained on the VKontakte corpus, which we also release to enable a fair comparison with our baselines in future work. This model achieved an F1 score of 0.728 in a 5-class classification setup. The dataset, the source code for the baseline classifiers, and the in-domain word embeddings are available at the project website2. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http: //creativecommons.org/licenses/by/4.0/. 1https://vk.com/about 2http://text-machine.cs.uml.edu/projects/rusentiment/ 756 2 Related Work Russian is generally not as well resourced as English, and that includes the …

Content truncated to 500 words