The Construction and Evaluation of the LEAFTOP Dataset of Automatically Extracted Nouns in 1480 Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5021–5028 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5021 The Construction and Evaluation of the LEAFTOP Dataset of Automatically Extracted Nouns in 1480 Languages Greg Baker, Diego Molla-Aliod Macquarie University 4 Research Park Drive gregory.baker2@hdr.mq.edu.au, diego.molla-aliod@mq.edu.au Abstract The LEAFTOP (language extracted automatically from thousands of passages) dataset consists of nouns that appear in multiple places in the four gospels of the New Testament. We use a naive approach — probabilistic inference — to identify likely translations in 1480 other languages. We evaluate this process and find that it provides lexiconaries with accuracy from 42% (Korafe) to 99% (Runyankole), averaging 72% correct across evaluated languages. The process translates up to 161 distinct lemmas from Koine Greek (average 159). We identify nouns which appear to be easy and hard to translate, language families where this technique works, and future possible improvements and extensions. The claims to novelty are: the use of a Koine Greek New Testament as the source language; using a fully-annotated manually-created grammatically parse of the source text; a custom scraper for texts in the target languages; a new metric for language similarity; a novel strategy for evaluation on low-resource languages. Keywords: leaftop, bible, corpus, Koine Greek, lexicon, low-resource languages Figure 1: Paraphrased vs literal translations of the New Testament into English 1. Introduction This paper discusses a large new dataset designed to be useful for tasks involving part-of-speech identifi- cation, grammar morphology and language similarity measures which was created by extracting vocabulary from Bible translations. It establishes a baseline of ac- curacy for target language noun extraction using sensi- ble naive techniques at scale, for the languages spoken by the majority of the world’s population. The resulting LEAFTOP dataset is both wide (1480 languages) and deep (160 nouns). For inflected lan- guages it contains forms for number (singular vs plural) for all nouns; for some nouns it can also supply gender Language Family Languages Evaluated Niger-Congo Fon; Guinea Kpelle; Igbo; Samia; Luganda; Mano; Runyankole; Swahili; Twi; Soga; Yoruba Afro-Asiatic Tunisian Arabic; Modern Standard Arabic; Moroccan Arabic; Chadian Arabic Dravidian Telugu Artificial Esperanto Arnhem Gunwinggu Austronesian Cebuano; Dobu; Hiligaynon; Hiri Motu; Kilivila; Nyindrou; Takia; Tagalog Trans-New Guinea Korafe; Melpa Indo-European Bengali; German; French; Hindi; Marathi; Sinhala; Urdu Nilo-Saharan Teso Table 1: The subset of the 1480 languages in the LEAFTOP dataset which have been evaluated, and the language families they are part of. and case variations. The Ethnologue (Eberhard et al., 2021) reports the ex- istence of 7,139 living languages. There are surpris- ingly few data sets which cover a large proportion of the world’s population that also have sufficient depth for machine learning techniques. There are only 82 nouns listed in the extended Swadesh 207 list; the ASJP database (Wichmann and Brown, 2020) covers only the Swadesh 100 and is therefore even smaller with only 46 nouns. This is a vocabulary that is too small for many machine learning techniques. …

Content truncated to 500 words