The DReaM Corpus: A Multilingual Annotated Corpus of Grammars for the World’s Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 878–884 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 878 The DReaM Corpus: A Multilingual Annotated Corpus of Grammars for the World’s Languages Shafqat Mumtaz Virk1, Harald Hammarstr¨om2, Markus Forsberg1, Søren Wichmann3 1Spr˚akbanken Text, Department of Swedish, University of Gothenburg 2Department of Linguistics and Philology, University of Uppsala 3Leiden University Centre for Linguistics, Leiden University 3Laboratory of Quantitative Linguistics, Kazan Federal University 3Beijing Advanced Innovation Center for Language Resources, Beijing Language University 1{shafqat.virk, markus.forsberg}@gu.se 2{harald.hammarstrom}@lingfil.uu.se 3{wichmannsoeren}@gmail.com Abstract There exist as many as 7000 natural languages in the world, and a huge number of documents describing those languages have been produced over the years. Most of those documents are in paper format. Any attempts to use modern computational techniques and tools to process those documents will require them to be digitized first. In this paper, we report a multilingual digitized version of thousands of such documents searchable through some well-established corpus infrastructures. The corpus is annotated with various meta, word, and text level attributes to make searching and analysis easier and more useful. Keywords: corpus, natural languages, grammatical descriptions, world’s languages 1. Introduction The diversity of the 7000 languages of the world represents an irreplaceable and abundant resource for understanding the unique communication system of our species (Evans and Levinson, 2009). All comparison and analysis of lan- guages departs from language descriptions — publications that contain facts about particular languages. The typi- cal examples of this genre are grammars and dictionaries (Hammarstr¨om and Nordhoff, 2011). Until recently, language descriptions were available in pa- per form only, with indexes as the only search aid. In the present era, digitization and language technology promise broader perspectives for readers of language descriptions. The first generation of enhanced search tools allow search- ing across many documents using basic markup and filters, and modern natural language processing (NLP) tools can take exploitation arbitrarily further. In this paper we de- scribe the collection, digitization, management and search infrastructure so far developed for a comprehensive collec- tion of language descriptions. The paper is organized as follows: Section 2. describes the collection and digitization process, while the statistics of the corpus are given in Section 3. The methods applied to do post-OCR corrections are explained in Section 4. Sec- tion 5., briefly describes the two corpus infrastructures to be followed by Section 6., which explains how the corpus can be accessed using those two infrastructures. 2. Collection and digitization Enumerating the extant set of language descriptions for the languages of the world is a non-trivial task. Thanks to the Glottolog project, this task is now complete in the sense that the most extensive description for every language is known (Hammarstr¨om and Nordhoff, 2011). These refer- ences, along with a large body for further entries for most languages, are included in the open-access bibliography of (Hammarstr¨om et al., 2019). A core subset of about 30,000 publications — including the most extensive description …

Content truncated to 500 words