RoBERTuito: a pre-trained language model for social media text in Spanish

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7235–7243 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7235 RoBERTuito: a pre-trained language model for social media text in Spanish Juan Manuel P´erez12, Dami´an A. Furman12, Laura Alonso Alemany3, Franco Luque23 1Universidad de Buenos Aires, 2CONICET, 3Universidad Nacional de C´ordoba {jmperez, dfurman}@dc.uba.ar, {francolq, alemany}@famaf.unc.edu.ar Abstract Since BERT appeared, Transformer language models and transfer learning have become state-of-the-art for natural language processing tasks. Recently, some works geared towards pre-training specially-crafted models for particular domains, such as scientific papers, medical documents, user-generated texts, among others. These domain-specific models have been shown to improve performance significantly in most tasks; however, for languages other than English, such models are not widely available. In this work, we present RoBERTuito, a pre-trained language model for user-generated text in Spanish, trained on over 500 million tweets. Experiments on a benchmark of tasks involving user-generated text showed that RoBERTuito outperformed other pre-trained language models in Spanish. In addition to this, our model has some cross-lingual abilities, achieving top results for English-Spanish tasks of the Linguistic Code-Switching Evaluation benchmark (LinCE) and also competitive performance against monolingual models in English Twitter tasks. To facilitate further research, we make RoBERTuito publicly available at the HuggingFace model hub together with the dataset used to pre-train it. Keywords: Pre-trained Language Models, BERT, Spanish 1. Introduction Pre-trained language models have become a basic building block in the area of natural language process- ing. In the last years, since the introduction of the trans- former architecture (Vaswani et al., 2017), they have been used in many other different natural language understanding tasks, outperforming previous models based on recurrent neural networks. BERT, GPT-2, and RoBERTa are some of the most well-known such tools. Most models are trained on large-scale corpora taken from news or Wikipedia, which are considered general enough to comprise a large part of the language. Some domains, however, are very specific and have their own vocabulary, jargon, or complex expressions. The med- ical or scientific domains, for example, use terms and concepts which are not found in a general corpus or oc- cur just a few times. In some other cases, words have specific meanings within a particular domain. Collo- quial language –as found in Twitter and other social networks– is more informal, with slang and other ex- pressions which rarely occur in Wikipedia. For these reasons, a number of pre-trained models have been created to handle these domains. SciBERT (Beltagy et al., 2019) and MedBERT (Rasmy et al., 2021) are examples of domain-specific models. For user-generated text, many models have been trained on Twitter for different languages. However, Span- ish lacks pre-trained models for user-generated text, or they are not easily available in the most popular model repositories, such as the HuggingFace model hub. This hinders the development of accurate applications for user-generated text in Spanish. In this paper, we present RoBERTuito, a large-scale transformer model for user-generated text trained on Spanish tweets. …

Content truncated to 500 words