Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of Yorùbá and Twi

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2754–2762 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2754 Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of Yor`ub´a and Twi Jesujoba O. Alabi⋆†‡ Kwabena Amponsah-Kaakyire⋆†‡ David I. Adelani‡∥ Cristina Espa˜na-Bonet†‡ †DFKI GmbH, Saarbr¨ucken, Germany ∥Spoken Language Systems (LSV), Saarland Informatics Campus, ‡Saarland University, Saarbr¨ucken, Germany {jesujoba oluwadara.alabi, kwabena.amponsah-kaakyire, cristinae}@dfki.de, didelani@lsv.uni-saarland.de Abstract The success of several architectures to learn semantic representations from unannotated text and the availability of these kind of texts in online multilingual resources such as Wikipedia has facilitated the massive and automatic creation of resources for multiple languages. The evaluation of such resources is usually done for the high-resourced languages, where one has a smorgasbord of tasks and test sets to evaluate on. For low-resourced languages, the evaluation is more difficult and normally ignored, with the hope that the impressive capability of deep learning architectures to learn (multilingual) representations in the high-resourced setting holds in the low-resourced setting too. In this paper we focus on two African languages, Yor`ub´a and Twi, and compare the word embeddings obtained in this way, with word embeddings obtained from curated corpora and a language-dependent processing. We analyse the noise in the publicly available corpora, collect high quality and noisy data for the two languages and quantify the improvements that depend not only on the amount of data but on the quality too. We also use different architectures that learn word representations both from surface forms and characters to further exploit all the available information which showed to be important for these languages. For the evaluation, we manually translate the wordsim-353 word pairs dataset from English into Yor`ub´a and Twi. We extend the analysis to contextual word embeddings and evaluate multilingual BERT on a named entity recognition task. For this, we annotate with named entities the Global Voices corpus for Yor`ub´a. As output of the work, we provide corpora, embeddings and the test suits for both languages. Keywords: Multilingual embeddings, Low-resource language, Yor`ub´a, and Twi 1. Introduction In recent years, word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) have been proven to be very useful for training downstream natural language processing (NLP) tasks. Moreover, contextual- ized embeddings (Peters et al., 2018; Devlin et al., 2019) have been shown to further improve the performance of NLP tasks such as named entity recognition, question an- swering, and text classification when used as word features because they are able to resolve ambiguities of word repre- sentations when they appear in different contexts. Differ- ent deep learning architectures such as multilingual BERT (Devlin et al., 2019), LASER (Artetxe and Schwenk, 2019) and XLM (Lample and Conneau, 2019) have proved suc- cessful in the multilingual setting. All these architectures learn the semantic representations from unannotated text, making them cheap given the availability of texts in online multilingual resources such as Wikipedia. However, the evaluation of such resources is usually done for …

Content truncated to 500 words