Building and curating conversational corpora for diversity-aware language science and technology

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1178–1192 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 Building and curating conversational corpora for diversity-aware language science and technology Andreas Liesenfeld, Mark Dingemanse * Centre for Language Studies Radboud University, Netherlands andreas.liesenfeld@ru.nl, mark.dingemanse@ru.nl Abstract We present an analysis pipeline and best practice guidelines for building and curating corpora of everyday conversation in diverse languages. Surveying language documentation corpora and other resources that cover 67 languages and varieties from 28 phyla, we describe the compilation and curation process, specify minimal properties of a unified format for interactional data, and develop methods for quality control that take into account turn-taking and timing. Two case studies show the broad utility of conversational data for (i) charting human interactional infrastructure and (ii) tracing challenges and opportunities for current ASR solutions. Linguistically diverse conversational corpora can provide new insights for the language sciences and stronger empirical foundations for language technology. Keywords: corpus creation and curation, conversation, interactional linguistics, linguistic typology, dialog systems, speech recognition 1. Introduction Language resources that capture language use in its natu- ral habitat of social interaction are rare despite the obvious merits of studying the very environment where we all learn language and use it everyday (Schegloff, 2006). There are multiple reasons for this. Linguists have been trained to look the other way when it comes to what is considered mere performance (Boeckx, 2010). Collecting this kind of data requires one to venture out of lab settings and other controlled environments (Enfield, 2013), and transcribing it is resource-intensive (Himmelmann, 2018). These ob- stacles are compounded by the fact that most NLP work focuses on a handful of well-studied languages (Joshi et al., 2020; Blasi et al., 2021). However, under the auspices of various language documentation projects, language re- * Both authors contributed equally. Figure 1: Languages currently featured in the dataset plotted by geographic location of (one of their) speech communities. Coordinates from Glottolog (Hammarstr¨om et al., 2021); full list in Appendix. sources have been collected in more and more communities across the world (Seifart et al., 2018), and these often in- clude at least some conversational data. We argue such cor- pora harbour important insights for language science and technology. In this paper we describe efforts to build an open and reproducible pipeline for collating and curating corpora of conversational speech. We demonstrate use of the pipeline for a growing collection of corpora covering at least 67 lan- guages of 28 phyla (Figure 1). Around 75% of the corpora are sourced from existing language documentation projects. The remaining 25% come from other language resource platforms made available to the research community. In- vestigating a larger slice of the world’s linguistic diversity can strengthen the empirical foundation of the language sci- ences and foster diversity-aware language technologies. 1178 Figure 2: Language and corpus size in minutes (log scale). with various layers of information. Many existing pipelines for working with corpora are built …

Content truncated to 500 words