EENLP: Cross-lingual Eastern European NLP Index

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2050–2057 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2050 EENLP: Cross-lingual Eastern European NLP Index Alexey Tikhonov*, Alex Malkhasov, Andrey Manoshin, George Dima, R´eka Cserh´ati, Md.Sadek Hossain Asif, Matt S´ardi Independent researcher, Germany; Financial University of Russia; MEPhI, Russia; University Politehnica of Bucharest, Romania; University of Szeged, Hungary; Notre Dame College, Dhaka, Bangladesh; Mozaik Education, Hungary {altsoph, realex1902, sqidde, andreigeorgedima, cserhatir, asifsadek509, sardi.matt}@gmail.com Abstract Motivated by the sparsity of NLP resources for Eastern European languages, we present a broad index of existing Eastern European language resources (90+ datasets and 45+ models) published as a github repository open for updates from the community. Furthermore, to support the evaluation of commonsense reasoning tasks, we provide hand-crafted cross-lingual datasets for five different semantic tasks (namely news categorization, paraphrase detection, Natural Language Inference (NLI) task, tweet sentiment detection, and news sentiment detection) for some of the Eastern European languages. We perform several experiments with the existing multilingual models on these datasets to define the performance baselines and compare them to the existing results for other languages. Keywords: cross-lingual, less-resourced languages, NLP 1. Introduction Recent multilingual Transformer-based language models – such as mBERT (Devlin et al., 2018), XLM-RoBERTa (Sanh et al., 2019), multilingual DistilBERT (Conneau et al., 2020), etc. – show impressive results on different text analysis tasks and their cross-lingual reasoning capabilities are still actively studied (Lauscher et al., 2020). For example, the mBERT model trained on 104 languages has shown high cross-lingual performance; however, such evaluations mostly focused on cross-lingual transfer within high-resource languages (Wu and Dredze, 2020). Commonsense reasoning is one of the key prob- lems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English: there are widely spoken languages that still did not receive the focus of the research community (Joshi et al., 2020). Cross-lingual transfer learning could be beneficial for such languages in solving both theoretical and practical tasks. One can speculate that such high-level reasoning tasks could be less affected by the language syntax’s specifics and low-level properties, so it can be effective to use a cross-lingual approach here. However, it was shown that the success of cross-lingual transfer learning depends on different factors such as the amount of shared vocabulary, explicit alignment of representa- tions across languages, size of pretraining corpora, etc (Doddapaneni et al., 2021). To get a better understand- ing of the importance of these factors, researchers need * The corresponding author is Alexey Tikhonov to leverage diverse and detailed datasets. There are a bunch of cross-lingual datasets al- ready, such as XGLUE (Liang et al., 2020), XCOPA (Ponti et al., 2020), XL-WiC (Raganato et al., 2020), XWINO (Tikhonov and Ryabinin, 2021), Wino-X (Emelin and Sennrich, 2021), XNLI (Conneau et al., 2018), XL-WSD (Pasini et al., 2021), XTREME-R (Ruder et al., 2021), BSNLP (Piskorski et al., 2019), MOROCO (Butnaru and Ionescu, 2019), etc., but most of …

Content truncated to 500 words