Reliable Baselines for Sentiment Analysis in Resource-Limited Languages: The Serbian Movie Review Dataset

Reliable Baselines for Sentiment Analysis in Resource-Limited Languages: The Serbian Movie Review Dataset Vuk Batanović, Boško Nikolić, Milan Milosavljević School of Electrical Engineering, University of Belgrade Bulevar kralja Aleksandra 73, 11120 Belgrade, Serbia E-mail: bv115045p@student.etf.bg.ac.rs, nbosko@etf.bg.ac.rs, mmilan@etf.bg.ac.rs Abstract Collecting data for sentiment analysis in resource-limited languages carries a significant risk of sample selection bias, since the small quantities of available data are most likely not representative of the whole population. Ignoring this bias leads to less robust machine learning classifiers and less reliable evaluation results. In this paper we present a dataset balancing algorithm that minimizes the sample selection bias by eliminating irrelevant systematic differences between the sentiment classes. We prove its superiority over the random sampling method and we use it to create the Serbian movie review dataset – SerbMR – the first balanced and topically uniform sentiment analysis dataset in Serbian. In addition, we propose an incremental way of finding the optimal combination of simple text processing options and machine learning features for sentiment classification. Several popular classifiers are used in conjunction with this evaluation approach in order to establish strong but reliable baselines for sentiment analysis in Serbian. Keywords: sentiment classification, sample selection bias, dataset balancing 1. Introduction Research on sentiment analysis has been copious, but so far mostly focused on English, a language with abundant resources. Relatively little attention has been paid to the specificities of sentiment analysis in languages in which resources are hard to find, or to the issues which only become pronounced under such conditions. Sentiment classification is commonly benchmarked on the movie review domain, and most reference datasets in English are balanced (Pang, Lee, & Vaithyanathan, 2002; Pang & Lee, 2004; Maas et al., 2011). Hence, in order to explore how classifiers behave across different languages similarly sized balanced datasets are required. Collecting data for sentiment analysis in a resource- limited setting requires making the most out of all available data sources. These can vary wildly in terms of size, class distribution, and various linguistic properties. Thus, the collected data is likely to be imbalanced and to suffer from high sample selection bias, as it is not representative of the whole population (Zadrozny, 2004; Bareinboim, Tian, & Pearl, 2014). A common approach to constructing balanced datasets out of an imbalanced starting collection is random sampling. This technique works well if the initial data is free from sample selection bias (Van Hulse, Khoshgoftaar, & Napolitano, 2007). If it is not, however, it allows the bias to remain in the final balanced dataset. Training and evaluating machine learning classifiers on such a dataset makes the classifiers less robust and the evaluation less reliable. Cross-validation performances are inflated, since the classifiers learn to rely on specious patterns in the dataset which are wholly incidental. Conversely, performances on independent test data are diminished, since test data does not necessarily suffer from the same bias as the training dataset. We propose an alternative dataset balancing algorithm which attempts to minimize the sample selection bias. It does so …

Content truncated to 500 words