Resources and Experiments on Sentiment Classification for Georgian

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1613–1621 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1613 Resources and Experiments on Sentiment Classification for Georgian Nicolas Stefanovitch, Jakub Piskorski, Sopho Kharazi Joint Research Centre, Polish Academy of Sciences, Piksel SRL nicolas.stefanovitch@ec.europa.eu, jpiskorski@gmail.com, sopho.kharazi@ext.ec.europa.eu Abstract This paper presents, to the best of our knowledge, the first ever publicly available annotated dataset for sentiment classification and semantic polarity dictionary for Georgian. We describe the characteristics of these resources and the process of their creation in detail. We also report the results of various experiments on the performance of both lexicon- and machine learning-based models for Georgian sentiment classification. We consider both three- (positive, neutral, negative) and four-tier (positive, neutral, negative, mixed) classifications. The machine learning models explored include, logistic regression, support vector machines (SVMs), and transformer-based models. We also explore approaches based on transfer learning and translation (into a well-supported language). The results obtained for Georgian are on a par with state-of-the-art results in sentiment classification for well studied languages when using training data of comparable size. Keywords: sentiment analysis, low-resourced language, linguistic resources, Georgian language, machine learning 1. Introduction In this paper we report on creating linguistic resources for the purpose of sentiment detection and classifica- tion for the Georgian language, and we compare var- ious knowledge- and machine learning-based models for this task. The main driver behind this work is an ex- tension of the Europe Media Monitor (EMM), a large- scale multilingual news aggregation and analysis en- gine (JRC, 2018) to the processing of Georgian texts. One important component of the NLP pipeline is sen- timent analysis, but there is only little prior work and scarce linguistic resources related to sentiment analy- sis for Georgian exist. Georgian is significantly under- resourced, making it harder to develop NLP applica- tions for this language. To address this, we developed some resources for performing sentiment analysis in a real-world media monitoring environment. The main contributions of our work are: • creating of a sentiment polarity dictionary for Georgian containing circa 2K base and 70K com- plex entries, ranked on a four-tier scale of expres- sion (very positive, positive, negative, very nega- tive); • creating the first ever publicly released annotated sentiment dataset of ca. 4K text snippets, manu- ally annotated on a four tier scale (positive, neu- tral, negative, mixed) by multiple annotators; • using these resources to evaluate lexicon- and state-of-the-art machine learning-based sentiment classification approaches. The machine learning (ML) models explored include, i.a., logistic regression, SVMs, and XLM-Roberta, a multilingual transformer-based model. In particular, we explore different approaches for transformer-based models: direct fine-tuning on the Georgian corpus, fine tuning on the corpus machine translated into En- glish and transfer learning from an already trained model. The experiments consider both three-tier (pos- itive, neutral, negative) and four-tier (positive, neu- tral, negative, mixed) classification. Finally, we per- form a detailed study of the Inter-Annotator Agreement (IAA), and evaluate the impact of …

Content truncated to 500 words