Czech Dataset for Cross-lingual Subjectivity Classification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1381–1391 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1381 Czech Dataset for Cross-lingual Subjectivity Classification Pavel Pˇrib´aˇn1,2, Josef Steinberger1,2 University of West Bohemia, Faculty of Applied Sciences, 1NTIS – New Technologies for the Information Society, 2Department of Computer Science and Engineering, Univerzitn´ı 2732/8, 301 00 Pilsen, Czech Republic {pribanp,jstein}@kiv.zcu.cz https://nlp.kiv.zcu.cz Abstract In this paper, we introduce a new Czech subjectivity dataset of 10k manually annotated subjective and objective sentences from movie reviews and descriptions. Our prime motivation is to provide a reliable dataset that can be used with the existing English dataset as a benchmark to test the ability of pre-trained multilingual models to transfer knowledge between Czech and English and vice versa. Two annotators annotated the dataset reaching 0.83 of the Cohen’s κ inter-annotator agreement. To the best of our knowledge, this is the first subjectivity dataset for the Czech language. We also created an additional dataset that consists of 200k automatically labeled sentences. Both datasets are freely available for research purposes. Furthermore, we fine-tune five pre-trained BERT-like models to set a monolingual baseline for the new dataset and we achieve 93.56% of accuracy. We fine-tune models on the existing English dataset for which we obtained results that are on par with the current state-of-the-art results. Finally, we perform zero-shot cross-lingual subjectivity classification between Czech and English to verify the usability of our dataset as the cross-lingual benchmark. We compare and discuss the cross-lingual and monolingual results and the ability of multilingual models to transfer knowledge between languages. Keywords: subjectivity, dataset, Czech, cross-lingual, classification, transformers, benchmark 1. Introduction Subjectivity classification (Wiebe et al., 1999) is one of the integral parts of sentiment analysis (opinion min- ing). Its basic purpose is to determine if a sentence or phrase is subjective or objective (Liu, 2012). It can be further used to improve other tasks such as polarity de- tection or information extraction (Wiebe et al., 1999; Pang and Lee, 2004). Nowadays, the subjectivity clas- sification is often used as a benchmark test (Zhao et al., 2015; Reimers and Gurevych, 2019; Wang et al., 2021; Bragg et al., 2021) in transfer learning to test abilities and language understanding of pre-trained BERT-like language models based on the Transformer architecture (Vaswani et al., 2017). The goal of subjectivity classification is to classify a sentence or a clause of the sentence as subjective or objective. Subjective sentences express personal feel- ings, views, beliefs or opinions and objective sentences hold or describe some factual information (Liu, 2012). Evaluation of the pre-trained models for transfer learn- ing is a crucial part of their development. For English, the well-known GLUE (Wang et al., 2018) and Super- GLUE (Wang et al., 2019) benchmarks are available. These benchmarks contain a set of diverse tasks that al- low a thorough evaluation of English pre-trained mod- els. For multilingual models such as mBERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020), …

Content truncated to 500 words