Similarity-Based Estimation of Word Cooccurrence Probabilities

Similarity-Based Estimation of Word Cooccurrence Probabilities Ido Dagan Fernando Pereira AT&T Bell Laboratories 600 Mountain Ave. Murray Hill, NJ 07974, USA dagan©research, att. com pereira©research, att. com Abstract In many applications of natural language processing it is necessary to determine the likelihood of a given word combination. For example, a speech recognizer may need to determine which of the two word combinations "eat a peach" and "eat a beach" is more likely. Statis- tical NLP methods determine the likelihood of a word combination according to its frequency in a training cor- pus. However, the nature of language is such that many word combinations are infrequent and do not occur in a given corpus. In this work we propose a method for es- timating the probability of such previously unseen word combinations using available information on "most sim- ilar" words. We describe a probabilistic word association model based on distributional word similarity, and apply it to improving probability estimates for unseen word bi- grams in a variant of Katz's back-off model. The similarity-based method yields a 20% perplexity im- provement in the prediction of unseen bigrams and sta- tistically significant reductions in speech-recognition er- ror. Introduction Data sparseness is an inherent problem in statistical methods for natural language processing. Such meth- ods use statistics on the relative frequencies of config- urations of elements in a training corpus to evaluate alternative analyses or interpretations of new samples of text or speech. The most likely analysis will be taken to be the one that contains the most frequent config- urations. The problem of data sparseness arises when analyses contain configurations that never occurred in the training corpus. Then it is not possible to estimate probabilities from observed frequencies, andsome other estimation scheme has to be used. We focus here on a particular kind of configuration, word cooccurrence. Examples of such cooccurrences include relationships between head words in syntactic constructions (verb-object or adjective-noun, for exam- ple) and word sequences (n-grams). In commonly used models, the probability estimate for a previously un- seen cooccurrence is a function of the probability esti- Lillian Lee Division of Applied Sciences Harvard University 33 Oxford St. Cambridge MA 02138, USA llee©das, harvard, edu mates for the words in the cooccurrence. For example, in the bigram models that we study here, the probabil- ity P(w21wl) of a conditioned word w2 that has never occurred in training following the conditioning word wl is calculated from the probability of w~, as estimated by w2's frequency in the corpus (Jelinek, Mercer, and Roukos, 1992; Katz, 1987). This method depends on an independence assumption on the cooccurrence of Wl and w2: the more frequent w2 is, the higher will be the estimate of P(w2[wl), regardless of Wl. Class-based and similarity-based models provide an alternative to the independence assumption. In those models, the relationship between given words is mod- eled by analogy with other words that are in some sense similar to the given ones. Brown et a]. (1992) suggest a class-based n-gram model in …

Content truncated to 500 words