RefCo and its Checker: Improving Language Documentation Corpora’s Reusability Through a Semi-Automatic Review Process

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2721–2729 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2721 RefCo and its Checker: Improving Language Documentation Corpora’s Reusability Through a Semi-Automatic Review Process Herbert Lange, Jocelyn Aznar Universität Hamburg, ZAS Berlin, herbert.lange@uni-hamburg.de, aznar@leibniz-zas.de Abstract The QUEST (QUality ESTablished) project aims at ensuring the reusability of audio-visual datasets (Wamprechtshammer et al., 2022) by devising quality criteria and curating processes. RefCo (Reference Corpora) is an initiative within QUEST in collaboration with DoReCo (Documentation Reference Corpus, Paschen et al. (2020)) focusing on language documentation projects. Previously, Aznar and Seifart (2020) introduced a set of quality criteria dedicated to documenting fieldwork corpora. Based on these criteria, we establish a semi-automatic review process for existing and work-in-progress corpora, in particular for language documentation. The goal is to improve the quality of a corpus by increasing its reusability. A central part of this process is a template for machine-readable corpus documentation and automatic data verification based on this documentation. In addition to the documentation and automatic verification, the process involves a human review and potentially results in a RefCo certification of the corpus. For each of these steps, we provide guidelines and manuals. We describe the evalu- ation process in detail, highlight the current limits for automatic evaluation and how the manual review is organized accordingly. Keywords: QUEST, reusability, quality checking, language resources, oral language, annotated corpora, language doc- umentation 1. Introduction With the decline of languages around the world (Hale et al., 1992), documenting endangered and minority lan- guages became one of the current goals of linguistic research. The aim is to mitigate the loss of linguistic data as much as possible by creating archived material of existing endangered or minority languages for which we would otherwise have no language data. A consequent development is the inception of language documentation as a linguistic discipline, with its ob- jectives and own methodology: recording and conserv- ing linguistic data for its future reuse. This discipline emerged from the descriptive tradition and adopted many of its practices and recommendations, in par- ticular from the Boasian school (Himmelmann, 1998; Michaelis, 2014). But archiving data is not sufficient to ensure its usefulness: a specific curation process must be devised to accommodate for each intended audience, for instance linguists or the speech community from which the corpus originated. One approach to make the effort of language documentation pay most is by adhering to the FAIR principles (Wilkinson and others, 2016): Findability, Accessibility, Interoperability, and Reusability. However, adhering to these principles is not always straightforward and there are many ways of achieving them. Our goal is to improve existing and fu- ture corpora and their reuse by providing a process that helps corpus submitters1 to refine their data. Our method is threefold: documentation of a corpus, evaluation of the corpus against its documentation, and finally discussion of the assessment results between the corpus submitter and a reviewer in order to improve, 1We focus …

Content truncated to 500 words