Czert – Czech BERT-like Model for Language Representation

Proceedings of Recent Advances in Natural Language Processing, pages 1326–1338 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_149 1326 Czert – Czech BERT-like Model for Language Representation Jakub Sido* and Ondˇrej Pražák* and Pavel Pˇribáˇn and Jan Pašek and Michal Seják and Miloslav Konopík {sidoj,ondfa,pribanp,pasekj,sejakm,konopik}@kiv.zcu.cz NTIS – New Technologies for the Information Society, Department of Computer Science and Engineering, Faculty of Applied Sciences, University of West Bohemia, Technická 8, 306 14 Plzeˇn Czech Republic Abstract This paper describes the training process of the first Czech monolingual language representa- tion models based on BERT and ALBERT ar- chitectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss proper- ties of monolingual and multilingual models based upon our results. We publish all the pre- trained and fine-tuned models freely for the re- search community. 1 Introduction Transfer learning and pre-trained word embed- dings became a crucial component for most Nat- ural Language Processing (NLP) models. Con- textualized methods (McCann et al., 2017; Peters et al., 2018; Howard and Ruder, 2018) overcame the initial context insensitive word embeddings ap- proaches (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017). (McCann et al., 2017; Peters et al., 2018). The word representa- tions generated by the named methods are usu- ally used as input features for other task-specific models that are further trained. Starting with the BERT (Devlin et al., 2018), the BERT-like mod- els (Lan et al., 2020; Liu et al.; Sanh et al., 2019; Yang et al., 2019) based on Transformer architec- ture (Vaswani et al., 2017), achieved a significant performance improvement in many NLP tasks (Raf- fel et al., 2019). These recent models are trained on a language model task or tasks that are closely related to it. Such pre-training allows them to cap- ture the general representation of language and text. The pre-trained models are then directly fine-tuned *Equal contribution. with specific data for a selected downstream task. The performance improvement of these models is paid by the vastly increased requirements (i.e., data and computational resources) for their training. The mentioned models are primarily trained for English. Recently, models for other, mostly larger, languages have been released, e.g., French (Martin et al., 2020; Le et al., 2019), Polish (Kłeczek, 2020), Turkish (Schweter, 2020), Russian1, Italian2, Ger- man2, Arabic (Safaya et al., 2020), but also for languages that are spoken by a relatively small num- ber of people, i.e., Romanian (Dumitrescu et al., 2020), Dutch (Vries et al., 2019) or Finish (Vir- tanen et al., 2019). There were also introduced multilingual models (Conneau and Lample, 2019; Conneau et al., 2020), that can be used for multiple languages at once but usually at the cost of lower performance in comparison to solely monolingual models (Martin et al., 2020; Virtanen et al., 2019; Dumitrescu et …

Content truncated to 500 words