The Johns Hopkins University Bible Corpus: 1600+ Tongues for Typological Exploration

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2884–2892 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2884 The Johns Hopkins University Bible Corpus: 1600+ Tongues for Typological Exploration Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky Center for Language and Speech Processing Johns Hopkins University (arya, rewicks, dlewis77, amueller, wswu, oadams, gnicola2, yarowsky)@jhu.edu, post@cs.jhu.edu Abstract We present findings from the creation of a massively parallel corpus in over 1600 languages, the Johns Hopkins University Bible Corpus (JHUBC). The corpus consists of over 4000 unique translations of the Christian Bible and counting. Our data is derived from scraping several online resources and merging them with existing corpora, combining them under a common scheme that is verse-parallel across all translations. We detail our effort to scrape, clean, align, and utilize this ripe multilingual dataset. The corpus captures the great typological variety of the world’s languages. We catalog this by showing highly similar proportions of representation of Ethnologue’s typological features in our corpus. We also give an example application: projecting pronoun features like clusivity across alignments to richly annotate languages which do not mark the distinction. Keywords: low-resource NLP, parallel corpus, typology, function words, Bible 1. Introduction I thank my God, I speak with tongues more than ye all. (1 Cor. 14:18; King James Version) The high water mark for “low-resource” in NLP has risen in recent years. Guzm´an et al. (2019) present “low-resource” translation datasets of half a million parallel sentences, and McCarthy et al. (2019) evaluate on the 600k-sentence WMT Romanian–English dataset as a low-resource benchmark. By contrast, this work is concerned with mere thousands of sentences within each language. The Bible is a short but multi-parallel text organized in a consistent structure across languages. It has several dozen named constituent books, each divided into chapters. The chapters are collections of verses, which roughly map onto sentences. The book, chapter, and verse pinpoint the par- ticular sequence of text. While certain books may not be considered ‘canon’ and translators may disagree on which verses to include, the structure is sufficiently parallel for sev- eral applications. Further, the Bible provides high coverage of core vocabulary. Our corpus currently spans 1611 diverse written languages, with constituents of more than 90 language families. These languages display staggering breadth in typological features and their combinations. Common typological features (such as SVO, SOV, or VOS word order) and rarer features like inclusive/exclusive pronoun marking are present. Many of the languages in this corpus display features that are not present in major European languages or other large corpora, which makes this resource beneficial to further the research of machine translation and morphological analysis. The collection, cleaning, and alignment of new languages and translations is an ongoing project. As new versions of the Bible are scraped and prepared, our corpus continues to grow. Family JHU ETHN JHU % ETHN % Niger–Congo 313 1542 19.43 20.63 …

Content truncated to 500 words