Hindi Urdu Machine Transliteration using Finite-State Transducers

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 537–544 Manchester, August 2008 Hindi Urdu Machine Transliteration using Finite-state Transducers M G Abbas Malik Christian Boitet GTALP, Laboratoire d’Informatique Grenoble Université Joseph Fourier, France abbas.malik@imag.fr, Christian.Boitet@imag.fr Pushpak Bhattacharyya Dept. of Computer Science and Engineering, IIT Bombay, India pb@cse.iitb.ac.in Abstract Finite-state Transducers (FST) can be very efficient to implement inter-dialectal transliteration. We illustrate this on the Hindi and Urdu language pair. FSTs can also be used for translation between sur- face-close languages. We introduce UIT (universal intermediate transcription) for the same pair on the basis of their com- mon phonetic repository in such a way that it can be extended to other languages like Arabic, Chinese, English, French, etc. We describe a transliteration model based on FST and UIT, and evaluate it on Hindi and Urdu corpora. 1 Introduction Transliteration is mainly used to transcribe a word written in one language in the writing sys- tem of the other language, thereby keeping an approximate phonetic equivalence. It is useful for MT (to create possible equivalents of unknown words) (Knight and Stall, 1998; Paola and San- jeev, 2003), cross-lingual information retrieval (Pirkola et al, 2003), the development of multi- lingual resources (Yan et al, 2003) and multilin- gual text and speech processing. Inter-dialectal translation without lexical changes is quite useful and sometimes even necessary when the dialects in question use different scripts; it can be achieved by transliteration alone. That is the case of HUMT (Hindi-Urdu Machine Transliteration) where each word has to be transliterated from Hindi to Urdu and vice versa, irrespective of its © 2008. Licensed under the Creative Commons Attri- bution-Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc- sa/3.0/). Some rights reserved. type (noun, verb, etc. and not only proper noun or unknown word). “One man’s Hindi is another man’s Urdu” (Rai, 2000). The major difference between Hindi and Urdu is that the former is written in Devana- gari script with a more Sanskritized vocabulary and the latter is written in Urdu script (derivation of Persio-Arabic script) with more vocabulary borrowed from Persian and Arabic. In contrast to the transcriptional difference, Hindi and Urdu share grammar, morphology, a huge vocabulary, history, classical literature, cultural heritage, etc. Hindi is the National language of India with 366 million native speakers. Urdu is the National and one of the state languages of Pakistan and India respectively with 60 million native speakers (Rahman, 2004). Table 1 gives an idea about the size of Hindi and Urdu. Native Speakers 2nd Language Speakers Total Hindi 366,000,000 487,000,000 853,000,000 Urdu 60,290,000 104,000,000 164,290,000 Total 426,290,000 591,000,000 1,017,000,000 Table 1: Hindi and Urdu speakers Hindi and Urdu, being varieties of the same language, cover a huge proportion of world’s population. People from Hindi and Urdu com- munities can understand the verbal expressions of each other but not the written expressions. HUMT is an effort to bridge this scriptural divide between India and Pakistan. Hindi and Urdu scripts are briefly introduced in section 2. Universal Intermediate Transcrip- …

Content truncated to 500 words