Transliteration Mining Using Large Training and Test Sets

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 243–252, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Transliteration Mining Using Large Training and Test Sets Ali El Kahki, Kareem Darwish, Ahmed Saad El Din Mohamed Abd El-Wahab Qatar Computing Research Institute Qatar Foundation, Doha, Qatar Faculty of Computers and Information, Cairo University, Cairo, Egypt ame2154@columbia.edu, kdarwish@qf.org.qa, ataei@qf.org.qa wahab@writeme.com Abstract Much previous work on Transliteration Mining (TM) was conducted on short parallel snippets using limited training data, and successful methods tended to favor recall. For such methods, increasing training data may impact precision and application on large comparable texts may impact precision and recall. We adapt a state-of-the-art TM technique with the best reported scores on the ACL 2010 NEWS workshop dataset, namely graph reinforcement, to work with large training sets. The method models observed character mappings between language pairs as a bipartite graph and unseen mappings are induced using random walks. Increasing training data yields more correct initial mappings but induced mappings become more error prone. We introduce parameterized exponential penalty to the formulation of graph reinforcement and we estimate the proper parameters for training sets of varying sizes. The new formulation led to sizable improvements in precision. Mining from large comparable texts leads to the presence of phonetically similar words in target and source texts that may not be transliterations or may adversely impact candidate ranking. To overcome this, we extracted related segments that have high translation overlap, and then we performed TM on them. Segment extraction produced significantly higher precision for three different TM methods. 1. Introduction Transliteration Mining (TM) is the process of finding transliterations in parallel or comparable texts of different languages. For example, given the Arabic-English word sequence pairs: ( ﻲﻟﺎھﮪﮬﻫ ﻚﻠﻤﻟاﺍ ﻲﺳﻼﺳ, Haile Selassie I of Ethiopia), successful TM would mine the transliterations: (ﻲﻟﺎھﮪﮬﻫ, Haile) and (ﻲﺳﻼﺳ, Selassie). TM has been shown to be effective in several Information Retrieval (IR) and Natural Language Processing (NLP) applications. For example, in cross language IR, TM was used to handle out-of-vocabulary query words by mining transliterations between words in queries and top n retrieved documents and then using transliterations to expand queries (Udupa et al., 2009a). In Machine Translation (MT), TM can improve alignment at training time and help enrich phrase tables with named entities that may not appear in parallel training data. More broadly, TM is a character mapping problem. Having good character mapping models can be beneficial in a variety of applications such as learning stemming models, learning spelling transformations between similar languages, and finding variant spellings of names (Udupa and Kumar, 2010b). TM has attracted interest in recent years with a dedicated evaluation in the ACL 2010 NEWS workshop. In that evaluation, TM was performed using limited training data, namely 1,000 parallel transliteration word-pairs, on short parallel text segments, namely cross-language Wikipedia titles which were typically a few words long. Since TM was performed on very short parallel segments, the chances that two …

Content truncated to 500 words