Writing System and Speaker Metadata for 2,800+ Language Varieties

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5035–5046 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5035 Writing System and Speaker Metadata for 2,800+ Language Varieties Daan van Esch, Tamar Lucassen, Sebastian Ruder, Isaac Caswell, Clara Rivera Google Research 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA {dvanesch,tlucassen,ruder,icaswell,rivera}@google.com Abstract We describe an open-source dataset providing metadata for about 2,800 language varieties used in the world today. Specifically, the dataset provides the attested writing system(s) for each of these 2,800+ varieties, as well as an estimated speaker count for each variety. This dataset was developed through internal research and has been used for analyses around language technologies. This is the largest publicly-available, machine-readable resource with writing system and speaker information for the world’s languages. We analyze the distribution of languages and writing systems in our data and compare it to their representation in current NLP. We hope the availability of this data will catalyze research in under-represented languages. Keywords: multilingual, low-resource, natural language processing 1. Introduction Today, language technologies are easily available in only a small minority of the world’s 7,000+ language varieties. For example, technologies like automatic speech recognition and neural machine translation are available from commercial vendors in about 100 lan- guage varieties; even keyboards and spell-checkers, which are relatively straightforward to develop, are only available in about 1,000–1,500 varieties (Mager et al., 2018; van Esch et al., 2019; Kuhn et al., 2020). Fortunately, the last few years have seen an im- pressive amount of activity towards making language technologies available in more languages, including community-driven initiatives (Ardila et al., 2020; ∀ et al., 2020; Mager et al., 2021; Mirzakhalov, 2021). At the same time, advances in self-supervised pre-training for text (Howard and Ruder, 2018; Devlin et al., 2019) and speech (Baevski et al., 2020) have led to the devel- opment of pretrained multilingual models (Conneau et al., 2020; Chung et al., 2021; Babu et al., 2021), sig- nificantly reducing the barriers to extending language technologies to new languages (Pfeiffer et al., 2020; Muller et al., 2021). Such technologies are supported by a blossoming ecosystem of easy-to-use open-source tooling and libraries, such as Hugging Face1, but also more specialized tools like KeyMan2 for keyboard de- velopment; the machine translation toolkit JoeyNMT (Kreutzer et al., 2019); and the speech recognition toolkit Elpis (Foley et al., 2018). A challenge facing the expansion of language technol- ogy to more languages is a lack of easily accessible, machine-readable metadata about all language varieties of interest. While there are a number of available re- sources such as Ethnologue (Eberhard et al., 2021) and Glottolog (Hammarstr¨om et al., 2021) that provide in- formation about the world’s languages, these resources are either not publicly accessible (Ethnologue) or do not provide information about speaker numbers nor 1https://huggingface.co/ 2https://keyman.com/ # of language Speaker Writing Open- varieties data system data source Wikipedia list 100    ISO 639-3 7,893    Glottolog 8,549    …

Content truncated to 500 words