The birth of Romanian BERT

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4324–4328 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4324 The birth of Romanian BERT Stefan Daniel Dumitrescu Independent Researcher Bucharest, Romania Andrei-Marius Avram Politehnica Univ. of Bucharest Bucharest, Romania {dumitrescu.stefan,avram.andreimarius,sampo.pyysalo}@gmail.com Sampo Pyysalo University of Turku Turku, Finland Abstract Large-scale pretrained language models have become ubiquitous in Natural Language Pro- cessing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus com- position and cleaning, the model training pro- cess, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process. 1 Introduction A revolution started in natural language processing (NLP) a few years ago, when the first Transformer- based model (Vaswani et al., 2017) demonstrated a significant increase in state-of-the-art results com- pared to previous neural approaches. The bidirec- tional BERT (Devlin et al., 2018) language model has since been widely adopted as the baseline for transformer models, and it has been successfully applied to a broad range of NLP tasks from stan- dard language modeling to question answering, text summarization, and machine translation. A number of papers have been dedicated to studying why and how this model performs so well, including comparison to classical NLP (Ten- ney et al., 2019), investigation of the newly in- troduced multi-head attention mechanism (Michel et al., 2019), and analyses of what BERT learns (Clark et al., 2019). A good recent review is pre- sented by Rogers et al. (2020). Following BERT, a number of variations of language models using Corpus Lines Words Size OPUS 55.1 M 635.0 M 3.8 GB OSCAR 33.6 M 1725.8 M 11 GB Wikipedia 1.5 M 60.5M 0.4 GB Total 90.2 M 2421.3 M 15.2 GB Table 1: Corpus statistics (post-cleaning). the transformer architecture have been introduced, including extended models such as XLM (Lample and Conneau, 2019) and XLNet (Yang et al., 2019) as well as more efficient ones like DistillBERT (Sanh et al., 2019), ALBERT (Lan et al., 2019), and ELECTRA (Clark et al., 2020). However, the vast majority of these studies have focused only on English models. While Google has released a multilingual BERT model trained on 100+ languages, only recently have monolin- gual models for other languages started to appear: FlauBERT for French (Le et al., 2019), BERTje for Dutch (de Vries et al., 2019) and FinBERT for Finnish (Virtanen et al., 2019). But none for Ro- manian, until now. While the multilingual BERT can be used also for Romanian, a monolingual model can bring an increase in accuracy that can be observed also in …

Content truncated to 500 words