Give your Text Representation Models some Love: the Case for Basque

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4781–4788 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4781 Give your Text Representation Models some Love: the Case for Basque Rodrigo Agerri1∗, I˜naki San Vicente2∗, Jon Ander Campos1∗, Ander Barrena1∗, Xabier Saralegi2, Aitor Soroa1, Eneko Agirre1 1 IXA Group, HiTZ Centre, University of the Basque Country (UPV/EHU), 2Elhuyar Foundation {rodrigo.agerri, jonander.campos, ander.barrena, a.soroa, e.agirre}@ehu.eus {i.sanvicente, x.saralegi}@elhuyar.eus Abstract Word embeddings and pre-trained language models allow to build rich representations of text and have enabled improvements across most NLP tasks. Unfortunately they are very expensive to train, and many small companies and research groups tend to use models that have been pre-trained and made available by third parties, rather than building their own. This is suboptimal as, for many languages, the models have been trained on smaller (or lower quality) corpora. In addition, monolingual pre-trained models for non-English languages are not always available. At best, models for those languages are included in multilingual versions, where each language shares the quota of substrings and parameters with the rest of the languages. This is particularly true for smaller languages such as Basque. In this paper we show that a number of monolingual models (FastText word embeddings, FLAIR and BERT language models) trained with larger Basque corpora produce much better results than publicly available versions in downstream NLP tasks, including topic classification, sentiment classification, PoS tagging and NER. This work sets a new state-of-the-art in those tasks for Basque. All benchmarks and models used in this work are publicly available. Keywords: Neural language representation models, Information Extraction, Less-resourced languages 1. Introduction Word embeddings and pre-trained language models allow to build rich representations of text and have enabled im- provements across most NLP tasks. The most success- ful models include word embeddings like FastText (Bo- janowski et al., 2017), and, more recently, pre-trained lan- guage models which can be used to produce contextual em- beddings or directly fine-tuned for each task. Good exam- ples of the later are character-based models like Flair (Ak- bik et al., 2018) and masked language models like BERT (Devlin et al., 2019). In many cases the teams that devel- oped the algorithms also release their models, which facil- itates both reproducibility and their application in down- stream tasks. Given that the models are expensive to train, many companies and research groups tend to use those models, rather than building their own. This could be suboptimal as, for many languages, the models have been trained on easily obtained public corpora, which tends to be smaller and/or of lower quality that other existing corpora. In addition, pre-trained language models for non-English languages are not always available. In that case, only mul- tilingual versions are available, where each language shares the quota of substrings and parameters with the rest of the languages, leading to a decrease in performance (Devlin et al., 2019). The chances for smaller languages, as for in- stance Basque, seem even direr, …

Content truncated to 500 words