When Harry Met Harri: Cross-lingual Name Spelling Normalization

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 391–399, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics When Harry Met Harri,
 and
: Cross-lingual Name Spelling Normalization Fei Huang , Ahmad Emami and Imed Zitouni IBM T. J. Watson Research Center 1101 Kitchawan Road Yorktown Heights, NY 10598 {huangfe, emami, izitouni}@us.ibm.com Abstract Foreign name translations typically include multiple spelling variants. These variants cause data sparseness problems, increase Out-of-Vocabulary (OOV) rate, and present challenges for machine translation, information extraction and other NLP tasks. This paper aims to identify name spelling variants in the target language using the source name as an anchor. Based on word- to-word translation and transliteration probabilities, as well as the string edit distance metric, target name translations with similar spellings are clustered. With this approach tens of thousands of high precision name translation spelling variants are extracted from sentence-aligned bilingual corpora. When these name spelling variants are applied to Machine Translation and Information Extraction tasks, improvements over strong baseline systems are observed in both cases. 1 Introduction Foreign names typically have multiple spelling variants after translation, as seen in the following examples: He confirmed that "al-Kharroub province is at the top of our priorities." …for the Socialist Progressive Party in upper Shuf and the Al- Kharrub region,… …during his tour of a number of villages in the region of Al- Kharub,… …Beirut and its suburbs and Iqlim al-Khurub,…


Such name spelling variants also frequently appear in other languages, such as
(bushi) /
(bushu) /
(buxi) (for Bush) in Chinese, and
 (sbrngfyld) /
  (sbryngfyld) /
  (sbrynjfyld) (for Springfield) in Arabic. These spelling variants present challenges for many NLP tasks, increasing vocabulary size and OOV rate, exacerbating the data sparseness problem and reducing the readability of MT output when different spelling variants are generated for the same name in one document. We address this problem by replacing each spelling variant with its corresponding canonical form. Such text normalization could potentially benefit many NLP tasks including information retrieval, information extraction, question answering, speech recognition and machine translation. Research on name spelling variants has been studied mostly in Information Retrieval research, especially in query expansion and cross-lingual IR. Baghat and Hovy (2007) proposed two approaches for spelling variants generation, based on the letters-to-phonemes mapping and Soundex algorithm (Knuth 1973). Raghaven and Allan (2005) proposed several techniques to group names in ASR output and evaluated their effectiveness in spoken document retrieval (SDR). Both approaches use a named entity extraction system to automatically identify names. For multi-lingual name spelling variants, Linden (2005) proposed to use a general edit distance metric with a weighted FST to find technical term translations (which were referred to as “cross-lingual spelling variants”). These 391 variants are typically translated words with similar stems in another language. Toivonen and colleagues (2005) proposed a two-step fuzzy translation technique to solve similar problems. Al-Onaizan and Knight (2002), Huang (2003) and Ji and Grishman (2007) investigated the general name entity translation problem, …

Content truncated to 500 words