Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2413–2423 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2413 Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset Brian Roark†, Lawrence Wolf-Sonkin†, Christo Kirov†, Sabrina J. Mielke‡, Cibu Johny†, Işın Demirşahin† and Keith Hall† †Google Research ‡Johns Hopkins University {roark,wolfsonkin,ckirov,cibu,isin,kbhall}@google.com sjmielke@jhu.edu Abstract This paper describes the Dakshina dataset, a new resource consisting of text in both the Latin and native scripts for 12 South Asian languages. The dataset includes, for each language: 1) native script Wikipedia text; 2) a romanization lexicon; and 3) full sentence parallel data in both a native script of the language and the basic Latin alphabet. We document the methods used for preparation and selection of the Wikipedia text in each language; collection of attested romanizations for sampled lexicons; and manual romanization of held-out sentences from the native script collections. We additionally provide baseline results on several tasks made possible by the dataset, including single word transliteration, full sentence transliteration, and language modeling of native script and romanized text. Keywords: romanization, transliteration, South Asian languages 1. Introduction Languages in South Asia – a region covering India, Pak- istan, Bangladesh and neighboring countries – are generally written with Brahmic or Perso-Arabic scripts, but are also often written in the Latin script, most notably for informal communication such as within SMS messages, WhatsApp, or social media. While the use of the Latin script as the means for inputting text (romanization) is relatively com- mon for other languages, South Asian languages lack com- mon standard romanization systems (such as pinyin for Chi- nese). In other words, when individuals use the Latin script to write in South Asian languages, they do not adhere to a system, i.e., there is no commonly used standard Latin script orthography in these languages. Rather, individuals generally use the Latin script to provide a rough phonetic transcription of the intended word, which can vary from in- dividual to individual due to any number of factors, includ- ing regional or dialectal differences in pronunciations, dif- fering conventions of transcription, or simple idiosyncrasy. For example, Wolf-Sonkin et al. (2019) present an example from the comment thread to a blog entry, which includes Hindi comments in both the native script Devanagari and in the Latin script, where the word भर्�ाचार is romanized var- iously as bhrastachar, bhrashtachar, barashtachaar and bharastachar, among others. The prevalence of the Latin script for writing these lan- guages has led to, among other things, work on text-entry systems for those languages in the Latin script, including systems that transliterate from Latin script input to the na- tive script of the language (Hellsten et al., 2017), as well as systems that produce text in the Latin script but provide model-based features such as auto-correction (Wolf-Sonkin et al., 2019). Note, however, that this sort of script vari- ance, beyond just complicating systems for text entry, is also a challenge for common natural language …

Content truncated to 500 words