An Effective and Robust Framework for Transliteration Exploration

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1332–1340, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP An Effective and Robust Framework for Transliteration Exploration Ea-Ee Jan, Niyu Ge IBM T.J. Watson Research Center Yorktown Height, NY 10598 {ejan,niyuge}@us.ibm.com Shih-Hsiang Lin, Berlin Chen# National Taiwan Normal University Taipei, Taiwan {shlin,berlin}@csie.ntnu.edu.tw Abstract Transliteration is the process of proper name translation based on pronunciation. It is an important process in many multilin- gual natural language tasks. A common and essential component of transliteration approaches is a verification mechanism that tests if the two names in different lan- guages are translations of each other. Al- though many transliteration systems have verification as a component, verification as a stand-alone problem is relatively new. In this paper, we propose a simple, effective and robust training framework for the task of verification. We show the many appli- cations of the verification techniques. Our proposed method can operate on both pho- nemic and orthographic inputs. Our best re- sults show that a simple, straightforward orthographic representation is sufficient and no complex training method is needed. It is effective because it achieves remark- able accuracies. It is robust because it is language-independent. We show that on Chinese and Korean our technique achieves equal error rate well below 1% and around 1% for Japanese using 2009 and 2010 NEWS transliteration generation share task dataset. Our results also show that the or- thographic system outperforms the phone- mic system. This is especially encouraging because the orthographic inputs are easier to generate and secondly, one does not need to resort to more complex training al- gorithm to achieve excellent results. This approach is integrated for proper name based cross lingual information retrieval without translation. 1 # This work was partially sponsored by "Aim for the Top University Plan" of National Taiwan Normal University and Ministry of Education, Taiwan. 1 Introduction Proper name transliteration is important in many multilingual natural language processing tasks, such as Machine Translation (MT), Cross Lingual Information Retrieval (CLIR), multilin- gual spoken document retrieval and transliteration mining. The research community has investigated automatic proper name transliteration generation. The best performance with 10 references is ap- proximately 70% for alphabet based edit distance error (Li et al., 2009). With one reference, the error rate can be as high as 50% (Meng et al., 2001; Virga and Khudanpur, 2003). If the error rate is measured using the whole proper name as a unit, the error rate will be even higher. Alternatively, method for transliteration veri- fication starts to draw attention in the research community. Given a pair of proper names in the source and target languages, the task is to decide whether they are transliterations of each other. This task is important for many applications. For ex- ample, in word alignment (Ittycheriah and Roukos, 2005), the unknown words are handled by comput- ing a similarity score with the words in the target language. A similarity score derived from translit- eration verification has been successfully …

Content truncated to 500 words