Predicting Sentiment of Polish Language Short Texts

Proceedings of Recent Advances in Natural Language Processing, pages 1321–1327, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_151 1321 Predicting Sentiment of Polish Language Short Texts Aleksander Wawer Institute of Computer Science Polish Academy of Sciences Jana Kazimierza 5 01-248 Warszawa, Poland axw@ipipan.waw.pl Julita Sobiczewska University of Warsaw Krakowskie Przedmie´scie 26/28 00-927 Warszawa, Poland j.sobiczewska96@gmail.com Abstract The goal of this paper is to use all available Polish language data sets to seek the best possible performance in supervised senti- ment analysis of short texts. We use text collections with labeled sentiment such as tweets, movie reviews and a sentiment treebank, in three comparison modes. In the first, we examine the performance of models trained and tested on the same text collection using standard cross-validation (in-domain). In the second we train mod- els on all available data except the given test collection, which we use for test- ing (one vs rest cross-domain). In the third, we train a model on one data set and apply it to another one (one vs one cross-domain). We compare wide range of methods including machine learning on bag-of-words representation, bidirec- tional recurrent neural networks as well as the most recent pre-trained architectures ELMO and BERT. We formulate conclu- sions as to cross-domain and in-domain performance of each method. Unsurpris- ingly, BERT turned out to be a strong per- former, especially in the cross-domain set- ting. What is surprising however, is solid performance of the relatively simple mul- tinomial Naive Bayes classifier, which per- formed equally well as BERT on several data sets. 1 Introduction Automated sentiment analysis usually involves training machine learning or deep learning models in supervised fashion. Typically, studies involve one data type and report high accuracy. For ex- ample, the seminal machine learning studies on IMDB movie reviews of (Pang et al., 2002) in- dicated accuracy over 80%. Recently, the accur- acy of a deep learning system reported on this data set exceeded 95% (Howard and Ruder, 2018). Similarly, sentence-level sentiment predictions ex- ceeded 95% accuracy on binary version of the popular Stanford Sentiment Treebank (abbrevi- ated as SST-2) (Liu et al., 2019). However, multiple studies indicate that the models trained on such data sets are in fact very far from being applicable universally. Machine and deep learning models tend to fit to specific type of texts and language. When applied to different language types in terms of both style and vocab- ulary, the performance drops sharply. This issue is often explored under the name of domain de- pendency or data set dependency. Several meth- ods have been proposed to address it, such as for example (Selmer et al., 2013). In another stream of related studies, authors considered the task of cross-domain sentiment analysis: adaptation of a model to target domain or text type that is different from the domain or text type that the model was trained on. Examples include (Peng et al., 2018). The goal of this article is to evaluate and com- pare supervised techniques of sentiment analysis of short texts …

Content truncated to 500 words