Syllabified Sequence-to-sequence attention for transliteration

Syllabified Sequence-to-sequence attention for transliteration G.Vyshnavi vgutta7@gatech.edu G.Sridevi dr1sridevi@vrsiddhartha.ac.in M.Ritesh ritesh.s.m@ril.com P. Krishna Reddy pkreddy@iiit.ac.in Abstract The problem of transliteration deals with the phonetic transcription of text from a source writing system into a target writing system. With the inception of neural net models like Sequence-to-sequence networks, transliteration has seen significant progress in the last decade. However, the accuracy of such systems is still far from ideal. This is made more appar- ent when the source text to be transliterated itself is entered incorrectly by intermediary users, further degrading the performance. In this paper, we propose Syllabified Sequence- to-Sequence network (Syll-S2S) towards im- proving the transliteration quality from Roman script to low-resource Indic scripts like Devana- gari. As part of this, we present the rules of Sonority sequencing principle to Devanagari and other Indic scripts. In addition, the pro- posed framework incorporates Elastic Search stack which maps incorrect transliterations to their existing reference transliterations for han- dling erroneous entries of source texts. Experi- ments demonstrate significant performance im- provement of the proposed framework with re- spect to the existing schemes. 1 Introduction English is one of the most widespread foreign lan- guages in India, a home to 22 official languages and more than 1000 dialects written in more than 14 different scripts. With the rapid advancements in worldwideweb and mobile devices, people in India create, share, tag and search multifaceted data multi- lingually but mostly using the Roman or Latin script (Chanda et al., 2010) across different mediums. The text written in a native language, but using a non- native script like Roman, mostly does not follow any standard spelling rule, but uses the orthography of the script based on pronunciation of the words. This process of phonetically transcribing a word or text from one writing system into the another writing sys- tem such that the pronunciation of the word remains same is called Transliteration. Transliteration is a part of Natural Language Pro- cessing (NLP) and has several useful applications; Cross language information retrieval, Machine trans- lation etc. It has wide ramifications in low-resource languages in general, where web presence is limited, specifically for Indian languages. A substantial por- tion of textual data being generated and queried upon the web belongs to the transliteration domain, thus containing a good amount of information and therefore needs to be studied. In this paper, we focus on transliteration from the most commonly used Roman script to native Indian scripts. Most of the major Indian language scripts are derived from the ancient Brahmi script and con- sequently are highly phonetic in nature. Hence, we primarily focus on the task of transliteration from Roman to Hindi. Hindi, an Indo-Aryan language, written in Devanagari, is the lingua-franca of In- dia. We therefore consider Hindi as the primary link for Roman to native Indian script transliteration, as the quantity of Hindi literature (especially online) is more than twice as in any other Indian language. For the Roman-Hindi transliteration task, the ear- liest methods are rule based approaches …

Content truncated to 500 words