Evaluation Benchmarks for Spanish Sentence Representations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6024–6034 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6024 Evaluation Benchmarks for Spanish Sentence Representations Vladimir Araujo1,2,6,7, Andr´es Carvallo1,6, Souvik Kundu3, Jos´e Ca˜nete4,6, Marcelo Mendoza5,6,7, Robert E. Mercer3, Felipe Bravo-Marquez4,6,7, Marie-Francine Moens2, Alvaro Soto1,6 1Pontificia Universidad Cat´olica de Chile, Santiago, Chile 2KU Leuven, Leuven, Belgium 3University of Western Ontario, London, Canada 4Department of Computer Science, University of Chile, Santiago, Chile 5Universidad T´ecnica Federico Santa Mar´ıa, Santiago, Chile 6 National Center for Artificial Intelligence Research (CENIA), Santiago, Chile 7 Millennium Institute for Foundational Research on Data (IMFD), Santiago, Chile vgaraujo@uc.cl Abstract Due to the success of pre-trained language models, versions of languages other than English have been released in recent years. This fact implies the need for resources to evaluate these models. In the case of Spanish, there are few ways to systematically assess the models’ quality. In this paper, we narrow the gap by building two evaluation benchmarks. Inspired by previous work (Conneau and Kiela, 2018; Chen et al., 2019), we introduce Spanish SentEval and Spanish DiscoEval, aiming to assess the capabilities of stand-alone and discourse-aware sentence representations, respectively. Our benchmarks include considerable pre-existing and newly constructed datasets that address different tasks from various domains. In addition, we evaluate and analyze the most recent pre-trained Spanish language models to exhibit their capabilities and limitations. As an example, we discover that for the case of discourse evaluation tasks, mBERT, a language model trained on multiple languages, usually provides a richer latent representation than models trained only with documents in Spanish. We hope our contribution will motivate a fairer, more comparable, and less cumbersome way to evaluate future Spanish language models. Keywords: discourse evaluation, language models, sentence evaluation, representation learning 1. Introduction Spanish is one of the most widely spoken languages. This fact has drawn the attention of the NLP com- munity to the development of resources for that lan- guage. As a result, some pre-trained Spanish lan- guage models (Etcheverry and Wonsever, 2016; Che et al., 2018; Ca˜nete et al., 2020; Guti´errez-Fandi˜no et al., 2021) have been released in recent years driven by self-supervised approaches. This proliferation of Spanish language models increases the need for anno- tated datasets to evaluate them. Consequently, Span- ish datasets for a wide variety of independent tasks have been proposed ( ´A. Garc´ıa Cumbreras et al., 2006; Cruz Mata et al., 2008; Artetxe et al., 2020; Huertas- Tato et al., 2022). However, little effort has been put into creating benchmarks that allow models to be eval- uated systematically and fairly. Recently, Ca˜nete et al. (2020) presented the GLUES benchmark, a compilation of natural language under- standing tasks in Spanish. This benchmark aims to evaluate the performance of models by fine-tuning them to a target task (Wang et al., 2018). In contrast, another methodology known as probing tasks aims to assess whether the resulting representations of the models are general-purpose (Conneau et al., 2018a). A probing task is …

Content truncated to 500 words