RobBERT: a Dutch RoBERTa-based Language Model

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255–3265 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3255 RobBERT: a Dutch RoBERTa-based Language Model Pieter Delobelle1 and Thomas Winters1 and Bettina Berendt1,2 1 Department of Computer Science, KU Leuven 2 Faculty of Electrical Engineering and Computer Science, TU Berlin firstname.lastname@kuleuven.be Abstract Pre-trained language models have been dom- inating the field of natural language process- ing in recent years, and have led to significant performance gains for various complex natu- ral language tasks. One of the most prominent pre-trained language models is BERT, which was released as an English as well as a multi- lingual version. Although multilingual BERT performs well on many tasks, recent studies show that BERT models trained on a single language significantly outperform the multi- lingual version. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train a Dutch version of BERT, we used RoBERTa, a robustly optimized BERT ap- proach, to train a Dutch language model called RobBERT. We measured its performance on various tasks as well as the importance of the fine-tuning dataset size. We also evaluated the importance of language-specific tokenizers and the model’s fairness. We found that Rob- BERT improves state-of-the-art results for var- ious tasks, and especially significantly outper- forms other models when dealing with smaller datasets. These results indicate that it is a powerful pre-trained model for a large vari- ety of Dutch language tasks. The pre-trained and fine-tuned models are publicly available to support further downstream Dutch NLP appli- cations. 1 Introduction The advent of neural networks in natural lan- guage processing (NLP) has significantly im- proved state-of-the-art results within the field. Ini- tially, recurrent neural networks and long short- term memory networks dominated the field. Later, the transformer model caused a revolution in NLP by dropping the recurrent part and only keeping attention mechanisms (Vaswani et al., 2017). The transformer model led to other popular language models, e.g. GPT-2 (Radford et al., 2018, 2019). BERT (Devlin et al., 2019) improved over previ- ous models and recurrent networks by allowing the system to learn from input text in a bidirec- tional way, rather than only from left-to-right or the other way around. This model was later re- implemented, critically evaluated and improved in the RoBERTa model (Liu et al., 2019). These large-scale attention-based models pro- vide the advantage of being able to solve NLP tasks by having a common, expensive pre-training phase, followed by a smaller fine-tuning phase. The pre-training happens in an unsupervised way by providing large corpora of text in the desired language. The second phase only needs a rela- tively small annotated dataset for fine-tuning to outperform previous popular approaches in one of a large number of possible language tasks. While language models are usually trained on English data, some multilingual models also ex- ist. These are usually trained on a large quan- …

Content truncated to 500 words