Frequency Estimates for Statistical Word Similarity Measures

Frequency Estimates for Statistical Word Similarity Measures Egidio Terra School of Computer Science University of Waterloo elterra@math.uwaterloo.ca C. L. A. Clarke School of Computer Science University of Waterloo claclark@plg2.uwaterloo.ca Abstract Statistical measures of word similarity have ap- plication in many areas of natural language pro- cessing, such as language modeling and in- formation retrieval. We report a comparative study of two methods for estimating word co- occurrence frequencies required by word sim- ilarity measures. Our frequency estimates are generated from a terabyte-sized corpus of Web data, and we study the impact of corpus size on the effectiveness of the measures. We base the evaluation on one TOEFL question set and two practice questions sets, each consisting of a number of multiple choice questions seek- ing the best synonym for a given target word. For two question sets, a context for the target word is provided, and we examine a number of word similarity measures that exploit this con- text. Our best combination of similarity mea- sure and frequency estimation method answers 6-8% more questions than the best results pre- viously reported for the same question sets. 1 Introduction Many different statistical tests have been proposed to measure the strength of word similarity or word associ- ation in natural language texts (Dunning, 1993; Church and Hanks, 1990; Dagan et al., 1999). These tests attempt to measure dependence between words by using statistics taken from a large corpus. In this context, a key assump- tion is that similarity between words is a consequence of word co-occurrence, or that the closeness of the words in text is indicative of some kind of relationship between them, such as synonymy or antonymy. Although word sequences in natural language are un- likely to be independent, these statistical tests provide quantitative information that can be used to compare pairs of co-occurring words. Also, despite the fact that word co-occurrence is a simple idea, there are a vari- ety of ways to estimate word co-occurrence frequencies from text. Two words can appear close to each other in the same document, passage, paragraph, sentence or fixed-size window. The boundaries for determining co- occurrence will affect the estimates and as a consequence the word similarity measures. Statistical word similarity measures play an impor- tant role in information retrieval and in many other natu- ral language applications, such as the automatic creation of thesauri (Grefenstette, 1993; Li and Abe, 1998; Lin, 1998) and word sense disambiguation (Yarowsky, 1992; Li and Abe, 1998). Pantel and Lin (2002) use word sim- ilarity to create groups of related words, in order to dis- cover word senses directly from text. Recently, Tan et al. (2002) provide an analysis on different measures of independence in the context of association rules. Word similarity is also used in language modeling ap- plications. Rosenfeld (1996) uses word similarity as a constraint in a maximum entropy model which reduces the perplexity on a test set by 23%. Brown et al. (1992) use a word similarity measure for language modeling in …

Content truncated to 500 words