Hindi-to-Urdu Machine Translation through Transliteration

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 465–474, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Hindi-to-Urdu Machine Translation Through Transliteration Nadir Durrani Hassan Sajjad Alexander Fraser Helmut Schmid Institute for Natural Language Processing University of Stuttgart {durrani,sajjad,fraser,schmid}@ims.uni-stuttgart.de Abstract We present a novel approach to integrate transliteration into Hindi-to-Urdu statisti- cal machine translation. We propose two probabilistic models, based on conditional and joint probability formulations, that are novel solutions to the problem. Our mod- els consider both transliteration and trans- lation when translating a particular Hindi word given the context whereas in pre- vious work transliteration is only used for translating OOV (out-of-vocabulary) words. We use transliteration as a tool for disambiguation of Hindi homonyms which can be both translated or translit- erated or transliterated differently based on different contexts. We obtain final BLEU scores of 19.35 (conditional prob- ability model) and 19.00 (joint probability model) as compared to 14.30 for a base- line phrase-based system and 16.25 for a system which transliterates OOV words in the baseline system. This indicates that transliteration is useful for more than only translating OOV words for language pairs like Hindi-Urdu. 1 Introduction Hindi is an official language of India and is writ- ten in Devanagari script. Urdu is the national lan- guage of Pakistan, and also one of the state lan- guages in India, and is written in Perso-Arabic script. Hindi inherits its vocabulary from Sanskrit while Urdu descends from several languages in- cluding Arabic, Farsi (Persian), Turkish and San- skrit. Hindi and Urdu share grammatical structure and a large proportion of vocabulary that they both inherited from Sanskrit. Most of the verbs and closed-class words (pronouns, auxiliaries, case- markers, etc) are the same. Because both lan- guages have lived together for centuries, some Urdu words which originally came from Arabic and Farsi have also mixed into Hindi and are now part of the Hindi vocabulary. The spoken form of the two languages is very similar. The extent of overlap between Hindi and Urdu vocabulary depends upon the domain of the text. Text coming from the literary domain like novels or history tend to have more Sanskrit (for Hindi) and Persian/Arabic (for Urdu) vocabulary. How- ever, news wire that contains text related to me- dia, sports and politics, etc., is more likely to have common vocabulary. In an initial study on a small news corpus of 5000 words, randomly selected from BBC1 News, we found that approximately 62% of the Hindi types are also part of Urdu vocabulary and thus can be transliterated while only 38% have to be translated. This provides a strong motivation to implement an end-to-end translation system which strongly relies on high quality transliteration from Hindi to Urdu. Hindi and Urdu have similar sound systems but transliteration from Hindi to Urdu is still very hard because some phonemes in Hindi have several or- thographic equivalents in Urdu. For example the “z” sound2 can only be written as whenever it occurs in …

Content truncated to 500 words