Measures of Distributional Similarity

Measures of Distributional Similarity Lillian Lee Department of Computer Science Cornell University Ithaca, NY 14853-7501 llee@cs, cornell, edu Abstract We study distributional similarity measures for the purpose of improving probability estima- tion for unseen cooccurrences. Our contribu- tions are three-fold: an empirical comparison of a broad range of measures; a classification of similarity functions based on the information that they incorporate; and the introduction of a novel function that is superior at evaluating potential proxy distributions. 1 Introduction An inherent problem for statistical methods in natural language processing is that of sparse data -- the inaccurate representation in any training corpus of the probability of low fre- quency events. In particular, reasonable events that happen to not occur in the training set may mistakenly be assigned a probability of zero. These unseen events generally make up a sub- stantial portion of novel data; for example, Es- sen and Steinbiss (1992) report that 12% of the test-set bigrams in a 75%-25% split of one mil- lion words did not occur in the training parti- tion. We consider here the question of how to es- timate the conditional cooccurrence probability P(v[n) of an unseen word pair (n, v) drawn from some finite set N x V. Two state-of-the-art technologies are Katz's (1987) backoff method and Jelinek and Mercer's (1980) interpolation method. Both use P(v) to estimate P(v[n) when (n, v) is unseen, essentially ignoring the identity of n. An alternative approach is distance-weighted averaging, which arrives at an estimate for un- seen cooccurrences by combining estimates for 25 cooccurrences involving similar words: 1 /P(v[n) ---- ~-~mES(n) sim(n, m)P(v[m) ~-]mES(n) sim(n, m) , (1) where S(n) is a set of candidate similar words and sim(n, m) is a function of the similarity between n and m. We focus on distributional rather than semantic similarity (e.g., Resnik (1995)) because the goal of distance-weighted averaging is to smooth probability distributions -- although the words "chance" and "probabil- ity" are synonyms, the former may not be a good model for predicting what cooccurrences the latter is likely to participate in. There are many plausible measures of distri- butional similarity. In previous work (Dagan et al., 1999), we compared the performance of three different functions: the Jensen-Shannon divergence (total divergence to the average), the L1 norm, and the confusion probability. Our experiments on a frequency-controlled pseu- doword disambiguation task showed that using any of the three in a distance-weighted aver- aging scheme yielded large improvements over Katz's backoff smoothing method in predicting unseen coocurrences. Furthermore, by using a restricted version of model (1) that stripped in- comparable parameters, we were able to empir- ically demonstrate that the confusion probabil- ity is fundamentally worse at selecting useful similar words. D. Lin also found that the choice of similarity function can affect the quality of automatically-constructed thesauri to a statis- tically significant degree (1998a) and the ability to determine common morphological roots by as much as 49% in precision (1998b). 1The term "similarity-based", which we have used previously, has …

Content truncated to 500 words