Preliminary Results on the Evaluation of Computational Tools for the Analysis of Quechua and Aymara

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5450–5459 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5450 Preliminary Results on the Evaluation of Computational Tools for the Analysis of Quechua and Aymara Marcelo-Yuji Himoro∗, Antonio Pareja-Lora∗,∗∗ * ATLAS (UNED - Universidad Nacional de Educaci´on a Distancia) ** Universidad de Alcal´a (UAH) Madrid, Spain mhimoro1@alumno.uned.es, antonio.parejal@uah.es Abstract This research has focused on evaluating the existing open-source morphological analyzers for two of the most widely spoken indigenous macrolanguages in South America, namely Quechua and Aymara. Firstly, we have evaluated their performance (precision, recall and F1 score) for the individual languages for which they were developed (Cuzco Quechua and Aymara). Secondly, in order to assess how these tools handle other individual languages of the macrolanguage, we have extracted some sample text from school textbooks and educational resources. This sample text was edited in the different countries where these macrolanguages are spoken (Colombia, Ecuador, Peru, Bolivia, Chile and Argentina for Quechua; and Bolivia, Peru and Chile for Aymara), and it includes their different standardized forms (10 individual languages of Quechua and 3 of Aymara). Processing this text by means of the tools, we have (i) calculated their coverage (number of words recognized and analyzed) and (ii) studied in detail the cases for which each tool was unable to generate any output. Finally, we discuss different ways in which these tools could be optimized, either to improve their performances or, in the specific case of Quechua, to cover more individual languages of this macrolanguage in future works as well. Keywords: natural language processing, annotation, minority language, Quechua, Aymara 1. Introduction Among the over 6,000 languages spoken in the world, only a minority of them have been provided so far with adequate computational resources, e.g. for their pro- cessing. Therefore, there is a clear need for new digital resources to be developed for these languages. This comprises, for instance, the creation of tools for their analysis or the improvement of the existing ones, if any. Another problem to be faced is linguistic varia- tion, which usually requires adapting or extending the existing resources for different varieties or related lan- guages, or even to develop new specific versions of these resources. One of the possible approaches towards the creation of new tools for the processing of these under-resourced languages would be to build some state-of-the-art ma- chine learning models (based on, e.g., deep learning). However, the scarcity of annotated and/or bilingual data that can be used for training is one of the limiting factors for the development of language processing and analyzing tools. Therefore, in most cases, improving, re-targeting and/or fine-tuning already developed tools (when they exist) seems an unavoidable task. This is in particular the reality of Quechua and Aymara, two languages which, despite being spoken by millions of people in South America, are still under-resourced lan- guages in this sense, due to the few resources developed for their processing to date. More specifically, they …

Content truncated to 500 words