Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933–4946 August 1–6, 2021. ©2021 Association for Computational Linguistics 4933 Are Multilingual Models the Best Choice for Moderately Under-Resourced Languages? A Comprehensive Assessment for Catalan Jordi Armengol-Estap´e, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, Marta Villegas Barcelona Supercomputing Center, Barcelona, Spain {jordi.armengol, casimiro.carrino, carlos.rodriguez1, ona.degibert, carme.armentano, aitor.gonzalez, maite.melero, marta.villegas}@bsc.es Abstract Multilingual language models have been a crucial breakthrough as they considerably re- duce the need of data for under-resourced languages. Nevertheless, the superiority of language-specific models has already been proven for languages having access to large amounts of data. In this work, we focus on Catalan with the aim to explore to what extent a medium-sized monolingual language model is competitive with state-of-the-art large multi- lingual models. For this, we: (1) build a clean, high-quality textual Catalan corpus (CaText), the largest to date (but only a fraction of the usual size of the previous work in monolingual language models), (2) train a Transformer- based language model for Catalan (BERTa), and (3) devise a thorough evaluation in a diver- sity of settings, comprising a complete array of downstream tasks, namely, Part of Speech Tagging, Named Entity Recognition and Clas- sification, Text Classification, Question An- swering, and Semantic Textual Similarity, with most of the corresponding datasets being cre- ated ex novo. The result is a new benchmark, the Catalan Language Understanding Bench- mark (CLUB), which we publish as an open resource, together with the clean textual cor- pus, the language model, and the cleaning pipeline. Using state-of-the-art multilingual models and a monolingual model trained only on Wikipedia as baselines, we consistently ob- serve the superiority of our model across tasks and settings. 1 Introduction Over the past decades, Natural Language Process- ing (NLP) has become a powerful technology that may be found behind many AI-based consumer products, such as voice assistants, automatic trans- lators, intelligent chatbots, etc. This undeniable success is somehow tarnished by the fact that most NLP resources and systems are available only for a small percentage of languages (Joshi et al., 2020). In contrast, most of the languages spoken in the world today, even some with millions of speak- ers, are left behind, both in the research and in the development of the technology. Recent break- throughs in deep learning, specifically the Trans- former architecture (Vaswani et al., 2017), have revolutionized the entire field and have opened the doors to powerful transfer learning and unsu- pervised techniques, making it possible for under- resourced languages to benefit -at least, partially- from the formidable advances taking place for En- glish. Transformed-based multilingual pre-trained models (Devlin et al., 2019; Conneau et al., 2020) soon showed an impressive increase of perfor- mance also for under-resourced languages, as they considerably reduce the amount of training data needed for a particular task. The question as to whether training language- specific models was worth the effort, given those impressive results, was more or less quickly re- solved, for …

Content truncated to 500 words