Are the Multilingual Models Better? Improving Czech Sentiment with Transformers

Proceedings of Recent Advances in Natural Language Processing, pages 1138–1149 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_128 1138 Are the Multilingual Models Better? Improving Czech Sentiment with Transformers Pavel Pˇrib´aˇn1,2and Josef Steinberger1,2 University of West Bohemia, Faculty of Applied Sciences, Czech Republic 1NTIS – New Technologies for the Information Society, 2Department of Computer Science and Engineering, {pribanp,jstein}@kiv.zcu.cz http://nlp.kiv.zcu.cz Abstract In this paper, we aim at improving Czech senti- ment with transformer-based models and their multilingual versions. More concretely, we study the task of polarity detection for the Czech language on three sentiment polarity datasets. We fine-tune and perform experi- ments with five multilingual and three mono- lingual models. We compare the monolingual and multilingual models’ performance, includ- ing comparison with the older approach based on recurrent neural networks. Furthermore, we test the multilingual models and their ability to transfer knowledge from English to Czech (and vice versa) with zero-shot cross-lingual classification. Our experiments show that the huge multilingual models can overcome the performance of the monolingual models. They are also able to detect polarity in another lan- guage without any training data, with perfor- mance not worse than 4.4 % compared to state- of-the-art monolingual trained models. More- over, we achieved new state-of-the-art results on all three datasets. 1 Introduction In recent years, BERT-like models (Devlin et al., 2019) based on the Transformer architecture (Vaswani et al., 2017) and generalized language models brought a significant improvement in per- formance in almost any NLP task (Raffel et al., 2020a), especially in English. Despite this fact, not much work has been recently done in senti- ment analysis for the Czech language with the lat- est Transformer models. We partly fill this gap by focusing on the Sentiment Classification task, also known as Polarity Detection. Polarity detection is a classification task where the goal is to assign a sentiment polarity to a given text. The positive, negative and neutral classes are usually used as the polarity labels. The polarity can also be defined with a different number of labels, i.e., fine-grained sentiment analysis (Liu, 2012). The models based on BERT were almost ex- clusively trained for English, limiting their us- age to other languages. Recently, however, their cross-lingual adaptions like mBERT (Devlin et al., 2019), mT5 (Xue et al., 2020), XLM (Conneau and Lample, 2019) or XLM-R (Conneau et al., 2020) emerged along with other non-English monolingual versions, for example, Czech (Sido et al., 2021), French (Martin et al., 2020; Le et al., 2019), Ara- bic (Safaya et al., 2020), Romanian (Dumitrescu et al., 2020), Dutch (Vries et al., 2019) or Finnish (Virtanen et al., 2019). Our motivation is to reveal the performance lim- its of the current SotA transformer-based models on the Czech polarity detection task, check the ability of the multilingual models to transfer knowledge between languages and unify the procedure and data that enable the correct future evaluation of this task. In this paper, we focus on the task of polarity de- tection applied on Czech text by comparing the per- formance of …

Content truncated to 500 words