A More Discerning and Adaptable Multilingual Transliteration Mechanism for Indian Languages

A More Discerning and Adaptable Multilingual Transliteration Mechanism for Indian Languages Harshit Surana Language Tech. Research Centre IIIT, Hyderabad, India surana.h@gmail.com Anil Kumar Singh Language Tech. Research Centre IIIT, Hyderabad, India anil@research.iiit.ac.in Abstract Transliteration is the process of transcribing words from a source script to a target script. These words can be content words or proper nouns. They may be of local or foreign ori- gin. In this paper we present a more dis- cerning method which applies different tech- niques based on the word origin. The tech- niques used also take into account the prop- erties of the scripts. Our approach does not require training data on the target side, while it uses more sophisticated techniques on the source side. Fuzzy string matching is used to compensate for lack of training on the target side. We have evaluated on two Indian lan- guages and have achieved substantially bet- ter results (increase of up to 0.44 in MRR) than the baseline and comparable to the state of the art. Our experiments clearly show that word origin is an important factor in achiev- ing higher accuracy in transliteration. 1 Introduction Transliteration is a crucial factor in Cross Lingual Information Retrieval (CLIR). It is also important for Machine Translation (MT), especially when the languages do not use the same scripts. It is the pro- cess of transforming a word written in a source lan- guage into a word in a target language without the aid of a resource like a bilingual dictionary. Word pronunciation is usually preserved or is modified ac- cording to the way the word should be pronounced in the target language. In simple terms, it means finding out how a source word should be written in the script of the target languages such that it is ac- ceptable to the readers of the target language. One of the main reasons of the importance of transliteration from the point of view of Natural Lan- guage Processing (NLP) is that Out Of Vocabulary (OOV) words are quite common since every lexi- cal resource is very limited in practical terms. Such words include named entities, technical terms, rarely used or ‘difficult’ words and other borrowed words, etc. The OOV words present a challenge to NLP ap- plications like CLIR and MT. In fact, for very close languages which use different scripts (like Hindi and Urdu), the problem of MT is almost an extension of transliteration. A substantial percentage of these OOV words are named entities (AbdulJaleel and Larkey, 2003; Davis and Ogden, 1998). It has also been shown that cross language retrieval performance (average precision) reduced by more than 50% when named entities in the queries were not transliterated (Larkey et al., 2003). Another emerging application of transliteration (especially in the Indian context) is for building in- put methods which use QWERTY keyboard for peo- ple who are more comfortable typing in English. The idea is that the user types Roman letters but the input method transforms them into letters of In- …

Content truncated to 500 words