Mining Transliterations from Wikipedia using Dynamic Bayesian Networks

Proceedings of Recent Advances in Natural Language Processing, pages 385–391, Hissar, Bulgaria, 12-14 September 2011. Mining Transliterations from Wikipedia using Dynamic Bayesian Networks Peter Nabende Alfa-Informatica, University of Groningen p.nabende@rug.nl Abstract Transliteration mining is aimed at building high quality multi-lingual named entity (NE) lexicons for improving performance in various Natural Language Processing (NLP) tasks in- cluding Machine Translation (MT) and Cross Language Information Retrieval (CLIR). In this paper, we apply two Dynamic Bayesian network (DBN)-based edit distance (ED) ap- proaches in mining transliteration pairs from Wikipedia. Transliteration identification re- sults on standard corpora for seven language pairs suggest that the DBN-based edit distance approaches are suitable for modeling translit- eration similarity. An evaluation on mining transliteration pairs from English-Hindi and English-Tamil Wikipedia topic pairs shows that they improve transliteration mining qual- ity over state-of-the-art approaches. 1 Introduction Transliteration mining is aimed at addressing the problem of unknown words in NLP applications of which named entities (NEs) constitute the high- est percentage. Currently, there is growing inter- est in using automated methods to harness large amounts of correct multi-lingual named entities (NEs) from various ever accumulating Web-based data resources such as newspaper websites and on- line encyclopedia (most notably Wikipedia) with the aim of improving word coverage and the effec- tiveness of NLP systems such as MT and CLIR. In this paper, we present the application of two DBN-based edit distance approaches in mining transliterations from Wikipedia. Our motivation to apply the DBN-based approaches for model- ing transliteration is founded on our observation of their successful application in NLP tasks (such as cognate identification (2005) and pronunci- ation classification (2005)) which have require- ments similar to transliteration mining. While transliteration mining currently demands new ap- proaches to complement or improve performance over existing methods, there was not yet any inves- tigation about the use of the DBN-based edit dis- tance approaches for mining transliteration pairs from ‘noisy’ data. The first approach is based on the classic HMM framework but models two ob- servation sequences (hence the name Pair HMM) instead of one observation sequence. The sec- ond approach is based on the representation and implementation of a memoryless stochastic trans- ducer (initially proposed by Ristad and Yianilos (1998) as a DBN model for learning string edit distance. We propose to evaluate the use of the two approaches in mining transliterations with re- spect to two subtasks. In the first subtask, we fol- low the same evaluation setup as that for a recent shared task on transliteration mining (Kumaran et al., 2010b) while using the same standard cor- pora. This first subtask ensures a comparison of the DBN model results against those for state-of- the-art systems that participated in the shared task since the DBN models are applied to the same standard transliteration corpora. In the second subtask, we investigate the possibility of applying proposed DBN models in mining transliterations from Wikipedia’s article content in addition to the Wikipedia paired topics. 2 Wikipedia Wikipedia is a free Web-based multi-lingual en- cyclopedia with …

Content truncated to 500 words