Extensions to Brahmic script processing within the Nisaba library: new scripts, languages and utilities

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6450–6460 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6450 Extensions to Brahmic script processing within the Nisaba library: new scripts, languages and utilities Alexander Gutkin†, Cibu Johny†, Raiomond Doctor‡∗, Lawrence Wolf-Sonkin◦, Brian Roark◦ Google Research †United Kingdom ‡India ◦United States {agutkin,cibu,raiomond,wolfsonkin,roark}@google.com Abstract The Brahmic family of scripts is used to record some of the most spoken languages in the world and is arguably the most diverse family of writing systems. In this work, we present several substantial extensions to Brahmic script functionality within the open-source Nisaba library of finite-state script normalization and processing utilities (Johny et al., 2021). First, we extend coverage from the original ten scripts to an additional ten scripts of South Asia and beyond, including some used to record endangered languages such as Dogri. Second, we augment the language layer so that scripts used by multiple languages in distinct ways can be processed correctly for more languages, such as the Bengali script when used for the low-resource language Santali. We document key changes to the finite-state engine required to support these new languages and scripts. Finally, we add new script processing utilities, including lightweight script-level reading normalization that (unlike existing visual normalization) does not preserve visual invariance, and a fixed-input transliteration mechanism specifically tailored to Brahmic text entry with ASCII characters. Keywords: finite-state, Brahmic writing systems, low resource, transliteration, normalization 1. Introduction Nisaba is an open-source library1 for processing the Brāhmī-derived scripts of South Asia (Johny et al., 2021). Nisaba uses finite-state algorithms that lever- age the fundamental organizing principle of such scripts — an orthographic syllable, or aks ̣ara (Coulmas, 1999; Bright, 1999) — to provide a range of script-specific utilities. The design was inspired by the formal alge- braic approach to Brahmic scripts advocated by Datta (1984) and substantially advanced by Sproat (2003), and employs finite-state automata (FSAs) and transduc- ers (FSTs) to efficiently and compactly represent sev- eral types of finite-state grammars for script normaliza- tion, transliteration and validation (Mohri et al., 2000; Gorman, 2016). In this paper, we present several re- cent important extensions to the library: the addition of many new scripts, some of which were sufficiently dis- tinct to require changes to aks ̣ara modeling; expansion of language (in addition to script) layer functionality, to permit language-specific processing when required; and new normalization and transliteration utilities. Nisaba originally covered ten major Brahmic scripts of South Asia: Bengali, Devanagari, Gujarati, Gur- mukhi, Kannada, Malayalam, Oriya (Odia), Sinhala, Tamil, and Telugu (Johny et al., 2021). In this work, we extend this set by an additional ten Brahmic scripts of South and Maritime South-East Asia: Baybayin (Tagalog), Lepcha, Limbu, Lontara (Bugis), Maithili (Tirhuta), Meetei Mayek, Prachalit (Newa), Sylheti Na- gri, Takri and Thaana. These scripts were selected be- cause they are either actively used to record the offi- ∗On contract from Optimum Solutions, Inc. 1https://github.com/google-research/nisaba/ cial regional languages (Vikash and Shubha, 2018; Mo- hamed, 2019) …

Content truncated to 500 words