KLEJ: Comprehensive Benchmark for Polish Language Understanding

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1191–1201 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1191 KLEJ: Comprehensive Benchmark for Polish Language Understanding Piotr Rybak1, Robert Mroczkowski1, Janusz Tracz1, and Ireneusz Gawlik1,2 1 ML Research at Allegro.pl, ul. Grunwaldzka 182, 60-166 Pozna´n, Poland {firstname.lastname}@allegro.pl 2 AGH University of Science and Technology Faculty of Computer Science, Electronics and Telecommunications, Department of Computer Science al. A. Mickiewicza 30, 30-059 Kraków, Poland Abstract In recent years, a series of Transformer-based models unlocked major improvements in gen- eral natural language understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the pro- posed methods. However, such benchmarks are available only for a handful of languages. To alleviate this issue, we introduce a com- prehensive multi-task benchmark for the Pol- ish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. We also in- troduce a new sentiment analysis task for the e-commerce domain, named Allegro Re- views (AR). To ensure a common evaluation scheme and promote models that generalize to different NLU tasks, the benchmark in- cludes datasets from varying domains and ap- plications. Additionally, we release HerBERT, a Transformer-based model trained specifi- cally for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an extensive evaluation, including sev- eral standard baselines and recently proposed, multilingual Transformer-based models. 1 Introduction The field of natural language understanding (NLU) experienced a major shift towards knowledge re- usability and transfer learning, a phenomenon well established in the field of computer vision. Such a shift was enabled by recent introduction of robust, general-purpose models suitable for fine-tuning, like ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018) and BERT (Devlin et al., 2019). These models significantly improved the state-of- the-art on numerous language understanding tasks. Since then, the progress accelerated significantly and the new Transformer-based (Vaswani et al., 2017) models are being published every month to claim the latest state-of-the-art performance. Such a pace of development would not be pos- sible without standardized and publicly available NLU evaluation benchmarks. Among the most pop- ular ones is the recently introduced GLUE (Wang et al., 2019a) consisting of a collection of tasks such as question answering, sentiment analysis, and textual entailment with texts coming from a di- verse set of domains. Some tasks come with nu- merous training examples, while others have lim- ited training data. On top of that, for some tasks, the training set represents a different domain than the test set. This promotes models that learn gen- eral language representations and are effective at transferring knowledge across various tasks and do- mains. The GLUE benchmark is constructed based on existing datasets, and its main contribution is the careful choice of tasks …

Content truncated to 500 words