BERTabaporu: Assessing a Genre-Specific Language Model for Portuguese NLP

Proceedings of Recent Advances in Natural Language Processing, pages 217–223 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_024 217 BERTabaporu: Assessing a Genre-specific Language Model for Portuguese NLP Pablo da Costa Matheus Pavan Wesley dos Santos School of Arts, Sciences and Humanities University of S˜ao Paulo Av Arlindo Bettio 1000, S˜ao Paulo, Brazil {pablo.costa,matheus.pavan,wesley.ramos.santos,samuel.caetano.silva,ivandre}@usp.br Samuel da Silva Ivandr´e Paraboni Abstract Transformer-based language models such as Bidirectional Encoder Representations from Transformers (BERT) are now mainstream in the NLP field, but extensions to languages other than English, to new domains and/or to more specific text genres are still in demand. In this paper we introduced BERTabaporu, a BERT language model that has been pre-trained on Twitter data in the Brazilian Portuguese lan- guage. The model is shown to outperform the best-known general-purpose model for this lan- guage in three Twitter-related NLP tasks, mak- ing a potentially useful resource for Portuguese NLP in general. 1 Introduction Transformer-based language models such as Bidi- rectional Encoder Representations from Transform- ers (BERT) (Devlin et al., 2019) are now main- stream in the NLP field, but extensions are still much in demand. New BERT models have been fine-tuned or built from scratch for many languages other than English (e.g., Maltese in (Micallef et al., 2022)), for specific domains (e.g., mental health in (Ji et al., 2022)), and for particular text genres (e.g., Twitter data in (Nguyen et al., 2020)). In the case of our target language - Brazilian Portuguese - the first and best-known representa- tive of this trend is BERTimbau, a general purpose BERT model built from a large collection of web documents (Souza et al., 2020). Despite its popu- larity among the Portuguese NLP community, how- ever, we notice that the particular kind of language employed on contemporary social media may be distinct from the training data considered in previ- ous work, making existing models potentially less suitable to handle recent social media text. In par- ticular, we notice that tweets are not only shorter and less structured than pieces of news, but words such as ’Covid’ may not be recognised by older language models. Based on these observations, we may ask whether Twitter-related applications may benefit from a more genre-specific model built from so- cial media data - as opposed to more standard or general text - perhaps along the lines of BERTweet, a Twitter-specific model for the English language described in (Nguyen et al., 2020), or the multi- lingual TwHIN-BERT (Zhang et al., 2022), also built from Twitter data. To shed light on this is- sue, this work introduces BERTabaporu, a BERT model built from a collection of 238 million tweets written by over 100 thousand unique Twitter users, and conveying over 2.9 billion tokens in total. The model has been evaluated in three Twitter-related text classification tasks, and its results are com- pared to those obtained by the general purpose, web-based BERTimbau in (Souza et al., 2020). In doing so, our goal is to introduce a novel resource for Portuguese NLP, and foster further applications …

Content truncated to 500 words