Evaluating Language Tools for Fifteen EU-official Under-resourced Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1866–1873 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1866 Evaluating Language Tools for Fifteen EU-official Under-resourced Languages Diego Alves, Gaurish Thakkar, Marko Tadić University of Zagreb, Faculty of Humanities and Social Sciences Ivana Lučića 3, 10000 Zagreb, Croatia dfvalio@ffzg.hr, gthakkar@m.ffzg.hr, marko.tadic@ffzg.hr Abstract This article presents the results of the evaluation campaign of language tools available for fifteen EU-official under-resourced languages. The evaluation was conducted within the MSC ITN CLEOPATRA action that aims at building the cross-lingual event-centric knowledge processing on top of the application of linguistic processing chains (LPCs) for at least 24 EU-official languages. In this campaign, we concentrated on three existing NLP platforms (Stanford CoreNLP, NLP Cube, UDPipe) that all provide models for under-resourced languages and in this first run we covered 15 under-resourced languages for which the models were available. We present the design of the evaluation campaign and present the results as well as discuss them. We considered the difference between reported and our tested results within a single percentage point as being within the limits of acceptable tolerance and thus consider this result as reproducible. However, for a number of languages, the results are below what was reported in the literature, and in some cases, our testing results are even better than the ones reported previously. Particularly problematic was the evaluation of NERC systems. One of the reasons is the absence of universally or cross-lingually applicable named entities classification scheme that would serve the NERC task in different languages analogous to the Universal Dependency scheme in parsing task. To build such a scheme has become one of our the future research directions. Keywords:​ language processing chains, under-resourced languages, evaluation 1. Introduction The goal of the Marie Skłodowska-Curie Innovative Training Network CLEOPATRA (Cross-lingual Event-centric Open Analytics Research Academy) is to 1 offer a unique interdisciplinary and cross-sectoral research and training programme, which will explore how we can begin to analyse and understand major events and their relations presented in digital media. It will facilitate advanced cross-lingual processing of textual and visual information related to key contemporary events at scale and will develop innovative methods for efficient and intuitive user access and interaction with multilingual information. This ambitious goal heavily relies on the necessary prerequisite: the establishment of online available language processing chains (LPCs) for at least 24 EU-official languages of which many are also under-resourced. The components of LPCs all belong to the BLARK-modules (Krauwer, 2003) and cover tokenisation, sentence splitting, PoS/MSD-tagging, lemmatisation, named entity recognition and classification (NERC) and dependency parsing. On the results of LPCs regularly applied to the sets of large publicly available text streams (news, blogs, posts, etc.), the CLEOPATRA knowledge processing pipeline will build upon and provide additional value such as knowledge graphs, entity linking with LOD, RDF triples, other Semantic Web processing, etc. We expect that one of the final results of CLEOPATRA relevant to the NLP community will be the …

Content truncated to 500 words