Transliteration Systems across Indian Languages Using Parallel Corpora

Transliteration Systems Across Indian Languages Using Parallel Corpora Rishabh Srivastava and Riyaz Ahmad Bhat Language Technologies Research Center IIIT-Hyderabad, India {rishabh.srivastava, riyaz.bhat}@research.iiit.ac.in Abstract Hindi is the lingua-franca of India. Al- though all non-native speakers can com- municate well in Hindi, there are only a few who can read and write in it. In this work, we aim to bridge this gap by building transliteration systems that could transliter- ate Hindi into at-least 7 other Indian lan- guages. The transliteration systems are de- veloped as a reading aid for non-Hindi read- ers. The systems are trained on the translit- eration pairs extracted automatically from a parallel corpora. All the transliteration sys- tems perform satisfactorily for a non-Hindi reader to understand a Hindi text. 1 Introduction India is home to languages from four language families namely Indo-Aryan, Dravidian, Aus- troasiatic and Tibeto-Burman. There are 22 offi- cial languages and more than 1000 dialects, which are written in more than 14 different scripts1 in this country. Hindi, an Indo-Aryan language, written in Devanagari, is the lingua-franca of India (Ma- sica, 1993, p. 6). Most Indians are orally profi- cient in Hindi while they lack a good proficiency in reading and writing it. In this work, we come up with transliteration systems, so that non-native speakers of Hindi don’t face a problem in reading Hindi script. We considered 7 Indian languages, including 4 Indo-Aryan (Punjabi, Gujarati, Urdu and Bengali) and 3 Dravidian (Telugu, Tamil and Malayalam) languages, for this task. The quantity of Hindi literature (especially online) is more than twice as in any other Indian language. There are approximately 107 newspapers2, 15 online news- papers3 and 94067 Wikipedia articles4 (reported 1http://en.wikipedia.org/wiki/Languages of India 2http://en.wikipedia.org/wiki/List of newspapers in India 3http://www.indiapress.org/ 4http://stats.wikimedia.org/EN India/Sitemap.htm in March 2013), which are published in Hindi. The transliteration systems will be helpful for non- Hindi readers to understand these as well as vari- ous other existing Hindi resources. As the transliteration task has to be done for 7 languages, a rule-based system would become very expensive. The cost associated with craft- ing exhaustive rule-sets for transliteration has al- ready been demostrated in works on Hindi-Punjabi (Goyal and Lehal, 2009), Hindi-Gujarati (Patel and Pareek, 2009) and Hindi-Urdu (Malik et al., 2009; Lehal and Saini, 2010). In this work, we have modelled the task of transliteration as a noisy channel model with minimum error rate training (Och, 2003). However, such a statistical mod- elling needs an ample amount of data for train- ing and testing. The data is extracted from an Indian language sentence aligned parallel corpora available for 10 Indian languages. These sen- tences are automatically word aligned across the languages. Since these languages are written in different scripts, we have used an Indian modifica- tion of the soundex algorithm (Russell and Odell, 1918) (henceforth Indic-Soundex) for a normal- ized language representation. Extraction of the transliteration pairs (two words having the similar pronunciation) is then followed by Longest Com- mon Subsequence (henceforth LCS) algorithm, a string similarity algorithm. The extracted pairs …

Content truncated to 500 words