TeDDi Sample: Text Data Diversity Sample for Language Comparison and Multilingual NLP

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1150–1158 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1150 TeDDi Sample: Text Data Diversity Sample for Language Comparison and Multilingual NLP Steven Moran1, Christian Bentz2, Ximena Gutierrez-Vasques3, Olga Sozinova3, Tanja Samardzic3 University of Neuchatel1, University of T¨ubingen2, URPP Language and Space, University of Zurich3 Neuchatel, Switzerland1, T¨ubingen, Germany2, Zurich, Switzerland3 steven.moran@unine.ch, chris@christianbentz.de {ximena.gutierrezvasques, olga.sozinova, tanja.samardzic}@uzh.ch Abstract We present the TeDDi sample, a diversity sample of text data for language comparison and multilingual Natural Language Processing. The TeDDi sample currently features 89 languages based on the typological diversity sample in the World Atlas of Language Structures. It consists of more than 20k texts and is accompanied by open-source corpus processing tools. The aim of TeDDi is to facilitate text-based quantitative analysis of linguistic diversity. We describe in detail the TeDDi sample, how it was created, data availability, and its added value through for NLP and linguistic research. Keywords: Corpora, Quantitative Typology, Language Diversity, Language Documentation 1. Introduction Following a long debate on the status of linguistic vari- ation, the need to move beyond a limited set of WEIRD languages (Henrich et al., 2010; Majid and Levin- son, 2010) is becoming widely recognized. A deeper and more complete understanding of language is being achieved through increased access to data from minor- ity and low-resource languages. The same tendency is visible in NLP, where new multilingual datasets are currently released at a fast pace. These datasets, used for training and testing language models, have become especially interesting in the context of cross-linguistic transfer with few-shot and or even zero-shot learning. The question of how to select languages to be included in multilingual samples is approached differently in the two fields. Linguists put more weight on representing a wide range of language families and areas, as well as structural features, collecting the data from grammars, and storing them in typological databases. Researchers in NLP, on the other hand, favor languages for which text data is readily available online. Here, we present the TeDDi sample, which constitutes an intersection between the two approaches. Namely, it contains text samples for a selection of languages from the World Atlas of Language Structures (WALS) – spanning diverse families and areas. Since this selec- tion is independent of text data availability, some lan- guages in the sample have rich resources (e.g., English, Russian, Japanese), while others are only documented through fieldwork (e.g., Rama, Kayardild, Bagirmi). The challenge with resource-rich languages is how to select the texts to be included in the sample. The chal- lenge with low-resource languages is entirely different, namely, finding, extracting and digitizing texts from low-resource sources, e.g., published grammars. In the current version, our resulting sample consists of more than 20K texts. It is accompanied with a set of open-source processing tools. Our goal is to facilitate the use of text-based quantitative methods for analyz- ing linguistic diversity in both linguistic research and NLP. We first …

Content truncated to 500 words