Comparing Two Techniques for Learning Transliteration Models Using a Parallel Corpus

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 129–137, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Comparing Two Techniques for Learning Transliteration Models Using a Parallel Corpus Hassan Sajjad Nadir Durrani Helmut Schmid Institute for Natural Language ProcessingAlexander Fraser University of Stuttgart {sajjad,durrani,schmid,fraser}@ims.uni-stuttgart.de Abstract We compare the use of an unsupervised transliteration mining method and a rule- based method to automatically extract lists of transliteration word pairs from a par- allel corpus of Hindi/Urdu. We build joint source channel models on the auto- matically aligned orthographic transliter- ation units of the automatically extracted lists of transliteration pairs resulting in two transliteration systems. We compare our systems with three transliteration systems available on the web, and show that our systems have better performance. We per- form an extensive analysis of the results of using both methods and show evidence that the unsupervised transliteration min- ing method is superior for applications requiring high recall transliteration lists, while the rule-based method is useful for obtaining high precision lists. 1 Introduction Urdu and Hindi are closely related languages which have a similar phonological, semantic and syntactic structure. Hindi is derived from San- skrit and Urdu is a mixture of Persian, Arabic, Turkish and Sanskrit. Both share closed class vo- cabulary which they inherit from Sanskrit. They differ however in the open class vocabulary and in the writing script used. Hindi is written in Devanagari script and borrows most of the open class vocabulary from Sanskrit. Urdu is written in Perso-Arabic script and borrows most of the open class vocabulary from Persian, Arabic, Turk- ish and Sanskrit. Both languages have lived to- gether for centuries and now share a large part of their vocabulary with each other. In an initial study on a small parallel corpus, we found that both languages share approximately 82% (tokens) and 62% (types) of the vocabulary. Transliterating overlapping words will help to bridge the script- ing gap between Hindi and Urdu. The remaining words must be converted into the other language with a bilingual dictionary which is beyond the scope of this work. In this paper, the term transliteration pair refers to a word pair where the words are translitera- tions of each other and the term transliteration unit refers to a character pair where the charac- ters are transliterations of each other. We are interested in building joint source channel mod- els for transliteration. Because we do not have a list of transliteration pairs to use as training data in building such a transliteration model, we use two methods to extract the list of transliteration pairs from a parallel corpus of Hindi/Urdu. The first method uses the transliteration mining algo- rithm of Sajjad et al. (2011) to automatically ex- tract transliteration pairs. This approach does not use any language specific knowledge. The sec- ond method uses handcrafted transliteration rules specific to the mapping between Hindi and Urdu to extract transliteration pairs. We automatically align the two lists of extracted transliteration pairs at …

Content truncated to 500 words