Detecting Transliterated Orthographic Variants via Two Similarity Metrics

Detecting Transliterated Orthographic Variants via Two Similarity Metrics Kiyonori Ohtake ATR SLT Keihanna Science City Kyoto 619-0288, Japan kiyonori.ohtake@atr.jp Youichi Sekiguchi Nagaoka Univ. of Tech. Nagaoka City, Niigata 940-2188, Japan sekiguti@nlp.nagaokaut.ac.jp Kazuhide Yamamoto Nagaoka Univ. of Tech. Nagaoka City, Niigata 940-2188, Japan yamamoto@fw.ipsj.or.jp Abstract We propose a detection method for or- thographic variants caused by translit- eration in a large corpus. The method employs two similarities. One is string similarity based on edit distance. The other is contextual similarity by a vec- tor space model. Experimental results show that the method performed a 0.889 F-measure in an open test. 1 Introduction This paper discusses a detection method for transliterated orthographic variants of foreign words. Transliteration of foreign words causes orthographic variants because there are several conditions required for transliterating. One may person transliterate to approximate pronunciation, whereas another one may conduct transliteration based on spelling. For example, the English word report can be transliterated into two Japanese words, “ ���� (ripooto)” and “ � ��� (re- pooto).” The former “ripooto” is based on an ap- proximation of its pronunciation, while “repooto” is transliterated from its spelling. In addition, several source languages can be transliterated. For instance, the English word virus corresponds to the Japanese words: “ �  (uirusu)” from Latin, “  �  (biirusu)” and “  �  (viirusu)” from German, while “   (bairasu)” or “    (vairasu)” are also possible as transliterations that approximate the English pronunciation. Moreover, some for- eign words end up in different forms in Japanese because of variation in English pronunciation; e.g., between British and American. For example, the English word body corresponds to two words: “    (bodi)” from British and “   (badi)” from American. One may think that if back-transliteration were done precisely, those variants would be back- transliterated into one word, and they would be recognized as variants. However, back- transliteration is known to be a very difficult task(Knight and Graehl, 1997). Not only Japanese but any language that has a phonetic spelling has this problem of transliter- ated orthographic variants. For example, English has variants for a Chinese proper noun as “Shan- haiguan,” “Shanhaikwan,” or “Shanhaikuan.” Nowadays, it is well recognized that ortho- graphic variant correction is an important process- ing step for achieving high performance in natu- ral language processing. In order to achieve ro- bust and reliable processing, we have to use many language resources: many types of corpora, dic- tionaries, thesauri, and so on. Orthographic vari- ants cause many mismatches at any stage of natural language processing. In addition, not only ortho- graphic variants, but also misspelled words tend to slip into a corpus. These words boost the perplex- ity of the corpus, and worsen the data sparseness problem. To date, several studies have tried to cope with this orthographic variant problem; however, they considered the problem in a relatively clean corpus that was well organized by natives of the target lan- guage. As with orthographic variants, …

Content truncated to 500 words