ALBETO and DistilBETO: Lightweight Spanish Language Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4291–4298 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4291 ALBETO and DistilBETO: Lightweight Spanish Language Models Jos´e Ca˜nete1,3, Sebasti´an Donoso1,3, Felipe Bravo-Marquez1,3,4, Andr´es Carvallo2,3, Vladimir Araujo2,3,4,5 1 Department of Computer Science, University of Chile, Santiago, Chile 2Pontificia Universidad Cat´olica de Chile, Santiago, Chile 3 National Center for Artificial Intelligence (CENIA), Santiago, Chile 4 Millennium Institute for Foundational Research on Data (IMFD), Santiago, Chile 5KU Leuven, Leuven, Belgium jcanete@dcc.uchile.cl Abstract In recent years there have been considerable advances in pre-trained language models, where non-English language versions have also been made available. Due to their increasing use, many lightweight versions of these models (with reduced parameters) have also been released to speed up training and inference times. However, versions of these lighter models (e.g., ALBERT, DistilBERT) for languages other than English are still scarce. In this paper we present ALBETO and DistilBETO, which are versions of ALBERT and DistilBERT pre-trained exclusively on Spanish corpora. We train several versions of ALBETO ranging from 5M to 223M parameters and one of DistilBETO with 67M parameters. We evaluate our models in the GLUES benchmark that includes various natural language understanding tasks in Spanish. The results show that our lightweight models achieve competitive results to those of BETO (Spanish-BERT) despite having fewer parameters. More specifically, our larger ALBETO model outperforms all other models on the MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets. However, BETO remains unbeaten for POS and NER. As a further contribution, all models are publicly available to the community for future research. Keywords: ALBERT, BERT, DistilBERT, Efficient Models, Language Models 1. Introduction The area of Natural Language Processing has gained a lot of interest in recent years due to the great success of deep neural networks in different tasks such as sen- timent analysis (Socher et al., 2013; Rosenthal et al., 2017; Mohammad et al., 2018), sentence (Warstadt et al., 2019) and document classification (Schwenk and Li, 2018; Carvallo et al., 2020), natural language in- ference (Williams et al., 2017; Conneau et al., 2018), among others. This success can be explained by a large number of trainable parameters of these models, a considerable amount of training data, and the processing capacity power available nowadays, allowing them to learn com- plex relationships and cast complex functions. One of these highly successful models is BERT (De- vlin et al., 2019). This model, based on a Transformer (Vaswani et al., 2017) encoder, is pre-trained with lots of unlabeled text data and fine-tuned to specific tasks. BERT, in its large version, has 330 million parame- ters. Moreover, one of the most larger models is GPT-3 (Brown et al., 2020) which has 175 billion parameters. A problem with these large models is the difficulty of getting them into production on real-time applications (web services) or hardware-restricted devices, such as mobile devices, because of their high memory and computation requirements. Although there have been efforts for solving this issue …

Content truncated to 500 words