Hypothesis Selection in Machine Transliteration: A Web Mining Approach

Hypothesis Selection in Machine Transliteration: A Web Mining Approach Jong-Hoon Oh and Hitoshi Isahara Computational Linguistics Group National Institute of Information and Communications Technology (NICT) 3-5 Hikaridai,Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {rovellia,isahara}@nict.go.jp Abstract We propose a new method of selecting hy- potheses for machine transliteration. We generate a set of Chinese, Japanese, and Ko- rean transliteration hypotheses for a given English word. We then use the set of translit- eration hypotheses as a guide to finding rel- evant Web pages and mining contextual in- formation for the transliteration hypotheses from the Web page. Finally, we use the mined information for machine-learning al- gorithms including support vector machines and maximum entropy model designed to select the correct transliteration hypothesis. In our experiments, our proposed method based on Web mining consistently outper- formed systems based on simple Web counts used in previous work, regardless of the lan- guage. 1 Introduction Machine transliteration has been a great challenge for cross-lingual information retrieval and machine translation systems. Many researchers have devel- oped machine transliteration systems that accept a source language term as input and then output its transliteration in a target language (Al-Onaizan and Knight, 2002; Goto et al., 2003; Grefenstette et al., 2004; Kang and Kim, 2000; Li et al., 2004; Meng et al., 2001; Oh and Choi, 2002; Oh et al., 2006; Qu and Grefenstette, 2004). Some of these have used the Web to select machine-generated transliteration hypotheses and have obtained promising results (Al- Onaizan and Knight, 2002; Grefenstette et al., 2004; Oh et al., 2006; Qu and Grefenstette, 2004). More precisely, they used simple Web counts, estimated as the number of hits (Web pages) retrieved by a Web search engine. However, there are several limitations imposed on the ability of Web counts to select a correct translit- eration hypothesis. First, the assumption that hit counts approximate the Web frequency of a given query usually introduces noise (Lapata and Keller, 2005). Moreover, some Web search engines disre- gard punctuation and capitalization when matching search terms (Lapata and Keller, 2005). This can cause errors if such Web counts are relied on to se- lect transliteration hypotheses. Second, it is not easy to consider the contexts of transliteration hypothe- ses with Web counts because Web counts are esti- mated based on the number of retrieved Web pages. However, as our preliminary work showed (Oh et al., 2006), transliteration or translation pairs often appear as parenthetical expressions or tend to be in close proximity in texts; thus context can play an im- portant role in selecting transliteration hypotheses. For example, there are several Chinese, Japanese, and Korean (CJK) transliterations and their counter- parts in a parenthetical expression, as follows. 1) �����1����2 (Adrienne1 Clarkson2) 2) ���–�1����–�2 (glucose1 oxidase2) 3) �� ��1 ��� �� �� �2 (diphenol1 oxidase2) Note that the subscripted numbers in all examples represent the correspondence between the English word and its CJK counterpart. These parentheti- cal expressions are very useful in selecting translit- 233 eration hypotheses because it is apparent that they are …

Content truncated to 500 words