Evaluation of Transfer Learning for Polish with a Text-to-Text Model

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4374–4394 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4374 Evaluation of Transfer Learning for Polish with a Text-to-Text Model Aleksandra Chrabrowa, Łukasz Dragan, Karol Grzegorczyk, Dariusz Kajtoch Mikołaj Koszowski, Robert Mroczkowski, Piotr Rybak Allegro ML Research at Allegro.pl, ul. Grunwaldzka 182, 60-166 Pozna´n, Poland {firstname.lastname}@allegro.pl Abstract We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, and question answering. In particular, since summarization and question answering lack benchmark datasets for the Polish language, we describe their construction and make them publicly available. Additionally, we present plT5 - a general-purpose text-to-text model for Polish that can be fine-tuned on various Natural Language Processing (NLP) tasks with a single training objective. Unsupervised denoising pre-training is performed efficiently by initializing the model weights with a multi-lingual T5 (mT5) counterpart. We evaluate the performance of plT5, mT5, Polish BART (plBART), and Polish GPT-2 (papuGaPT2). The plT5 scores top on all of these tasks except summarization, where plBART is best. In general (except for summarization), the larger the model, the better the results. The encoder-decoder architectures prove to be better than the decoder-only equivalent. Keywords: benchmark, corpus, evaluation, NLU, NLG, T5, Polish language, summarization, 1. Introduction Recent years have brought significant progress in nat- ural language understanding (NLU) and natural lan- guage generation (NLG). Transformer architecture en- abled efficient training of large-scale language mod- els (Radford et al., 2019) and language understand- ing models (Devlin et al., 2019; Liu et al., 2019). On the other hand, transfer learning, which has been used in representation learning for years (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019), has finally been successfully applied to text-to-text problems as well (Raffel et al., 2020; Lewis et al., 2020). The text-to-text framework takes text as in- put and produces new text as output. This unified view enables the use of the same architecture, train- ing procedure, and decoding process for many NLP tasks such as classification, machine translation, sum- marization, and question answering, to name a few. In addition, Raffel et al. (2020) demonstrated that the simplicity of this approach combined with scale could achieve state-of-the-art results on many bench- mark datasets, which makes it even more attractive. Their T5 model was available only for English lan- guage, but more recently pre-trained multi-lingual ar- chitectures (Liu et al., 2020; Xue et al., 2021; Nagoudi et al., 2021a) and non-English counterparts (Carmo et al., 2020; Husein, 2018) were released. Multiple publi- cations show that models targeted for specific language perform better than multi-lingual one (Martin et al., 2020; Le et al., 2020; Chan et al., 2020; Mroczkowski et al., 2021; Virtanen et al., 2019; Nagoudi et al., 2021b). Moreover, specialized architectures are typi- cally smaller due to significantly reduced vocabulary size, and they can be trained efficiently via transfer …

Content truncated to 500 words