Automatic Construction of Weighted String Similarity Measures

Automatic Construction of Weighted String Similarity Measures JSrg Tiedemann Department of Linguistics Uppsala University joerg@stp.ling, uu.se Abstract String similarity metrics are used for several pur- poses in text-processing. One task is the extraction of cognates from bilingual text. In this paper three approaches to the automatic generation of language dependent string matching functions are presented. 1 Introduction String similarity metrics are extensively used in the processing of textual data for several purposes such as the detection and correction of spelling errors (Kukich, 1992), for sentence and word alignments (Church, 1993; Simard et al., 1992; Melamed, 1995), and the extraction of information from monolingnal as well as multi-lingual text (Resnik and Melamed, 1997; Borin, 1998; Tiedemann, 1998a). One im- portant task is the identification of so-called cog- nates, token pairs with a significant similarity be- tween them, in bilingual text. A commonly used technique for measuring string similarity is to look for the longest common subse- quence (LCS) of characters in two strings; the char- acters in this sequence do not necessarily need to be contiguous in the original strings (Wagner and Fis- cher, 1974; Stephen, 1992). The length of the LCS is usually divided by the length of the longer string of the two original tokens in order to obtain a normal- ized value. This score is called the longest common subsequence ratio- LCSR (Melamed, 1995). However, when it comes to different languages, a simple comparison of characters is usually not sat- isfactory to indicate the total correspondence be- tween words. Different languages tend to modify loan words derived from the same origin in different ways. Swedish and English are an example for two languages with a close etymological relation but a different way of spelling for a large set of cognates. The spelling usually follows certain language specific rules, e.g. the letter 'c' in English words corresponds to the letter 'k' in Swedish in most cases of cognates. Rules like this can be used for the recognition of cognates from specific language pairs. In this pa- per three approaches to the automatic generation of language pair specific string matching functions axe introduced. They include comparisons at the level of characters and n-grams with dynamic length. All the three approaches presume linguistic sim- ilarities between two languages. In this study they were applied to word pairs from a Swedish/English text corpus and experimental results are presented for each of them. 2 Resources Two types of textual resources were used in this study: * reference lexicons for the automatic generation of string matching functions • bilingual word pairs to be investigated with re- gard to string similarity A collection of bilingual word pairs is easy to produce. Similarity metrics should be applicable to every possible word pair from this set. How- ever, some restrictions can be imposed on the choice of appropriate pairs. In this study, all word pairs were derived from sentence aligned corpora of tech- nical texts which were collected in the PLUG corpus (Tiedemann, 1998b) as part of …

Content truncated to 500 words