RoBERT – A Romanian BERT Model

Proceedings of the 28th International Conference on Computational Linguistics, pages 6626–6637 Barcelona, Spain (Online), December 8-13, 2020 6626 RoBERT – A Romanian BERT Model Mihai Masala Stefan Ruseti University Politehnica of Bucharest 313 Splaiul Independentei 060042 Bucharest, Romania mihaimasala@gmail.com {stefan.ruseti, mihai.dascalu}@upb.ro Mihai Dascalu Abstract Deep pre-trained language models tend to become ubiquitous in the field of Natural Language Processing (NLP). These models learn contextualized representations by using a huge amount of unlabeled text data and obtain state of the art results on a multitude of NLP tasks, by enabling efficient transfer learning. For other languages besides English, there are limited options of such models, most of which are trained only on multi-lingual corpora. In this paper we introduce a Romanian-only pre-trained BERT model – RoBERT – and compare it with different multi- lingual models on seven Romanian specific NLP tasks grouped into three categories, namely: sentiment analysis, dialect and cross-dialect topic identification, and diacritics restoration. Our model surpasses the multi-lingual models, as well as a another mono-lingual implementation of BERT, on all tasks. 1 Introduction In the last decade, Recurrent Neural Networks (RNNs) based on LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Chung et al., 2014) cells represented the basis of state of the art methods for a wide range of Natural Language Processing (NLP) tasks (Bahdanau et al., 2015; Wang and Tan, 2016; Mehri and Carenini, 2017; Wang and Jiang, 2017). In general, RNNs make great use of pre-trained word em- beddings such as word2vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014). Word embeddings are usually computed using specialized neural networks trained in an unsupervised manner, and learn for each word a single vector representation. Recently, a paradigm-shift in the NLP community occurred: word embeddings were replaced by large-scale pre-trained language models that compute contextual embeddings (i.e., output embeddings depend on the entire sequence). Transformer (Vaswani et al., 2017) has quickly become the building block of multiple state of the art architectures such as GPT (Radford et al., 2018; Radford et al., 2019; Brown et al., 2020), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), or XLNET (Yang et al., 2019). Vaswani et al. (2017) propose the usage of multiple-head self-attention blocks, instead of the more classical recurrent approach, to model long-range sequence interactions. Replacing the sequential recurrent neural network with self-attention modules allows for easy parallelization, thus ensuring faster training on large-scale architectures. Large-scale transformers have the advantage of a single computationally expensive phase (pre-training), followed by an easy and fast fine-tuning phase, specific for each task. While transformer models have quickly become the standard approach for NLP tasks, the vast majority of studies have been performed on English. For other languages, the options are rather limited: either pre-train an entire model on the preferred language, or use a multi-lingual model trained on several languages. Two multi-lingual models stand out: multi-lingual BERT (Devlin et al., 2019), which is a BERT-base model trained on 104 languages, and XLM-RoBERTa (Conneau et al., 2020), which is …

Content truncated to 500 words