Contextual Word Similarity and Estimation From Sparse Data

CONTEXTUAL WORD SIMILARITY AND ESTIMATION FROM SPARSE DATA Ido Dagan AT•T Bell Laboratories 600 Mountain Avenue Murray Hill, NJ 07974 dagan@res earch, art. tom Shaul Marcus Computer Science Department Technion Haifa 32000, Israel shaul@cs, t echnion, ac. il $haul Markovitch Computer Science Department Technion Haifa 32000, Israel shaulm@cs, t echnion, ac. il Abstract In recent years there is much interest in word cooccurrence relations, such as n-grams, verb- object combinations, or cooccurrence within a limited context. This paper discusses how to estimate the probability of cooccurrences that do not occur in the training data. We present a method that makes local analogies between each specific unobserved cooccurrence and other cooccurrences that contain simi- lar words, as determined by an appropriate word similarity metric. Our evaluation sug- gests that this method performs better than existing smoothing methods, and may provide an alternative to class based models. 1 Introduction Statistical data on word cooccurrence relations play a major role in many corpus based approaches for natural language processing. Different types of cooccurrence relations are in use, such as cooc- currence within a consecutive sequence of words (n-grams), within syntactic relations (verb-object, adjective-noun, etc.) or the cooccurrence of two words within a limited distance in the context. Sta- tistical data about these various cooccurrence rela- tions is employed for a variety of applications, such as speech recognition (Jelinek, 1990), language gen- eration (Smadja and McKeown, 1990), lexicogra- phy (Church and Hanks, 1990), machine transla- tion (Brown et al., ; Sadler, 1989), information retrieval (Maarek and Smadja, 1989) and various disambiguation tasks (Dagan et al., 1991; Hindle and Rooth, 1991; Grishman et al., 1986; Dagan and Itai, 1990). A major problem for the above applications is how to estimate the probability of cooccurrences that were not observed in the training corpus. Due to data sparseness in unrestricted language, the ag- gregate probability of such cooccurrences is large and can easily get to 25% or more, even for a very large training corpus (Church and Mercer, 1992). Since applications often have to compare alterna- tive hypothesized cooccurrences, it is important to distinguish between those unobserved cooccur- rences that are likely to occur in a new piece of text and those that are not. These distinctions ought to be made using the data that do occur in the cor- pus. Thus, beyond its own practical importance, the sparse data problem provides an informative touchstone for theories on generalization and anal- ogy in linguistic data. The literature suggests two major approaches for solving the sparse data problem: smoothing and class based methods. Smoothing methods es- timate the probability of unobserved cooccurrences using frequency information (Good, 1953; Katz, 1987; Jelinek and Mercer, 1985; Church and Gale, 1991). Church and Gale (Church and Gale, 1991) show, that for unobserved bigrams, the estimates of several smoothing methods closely agree with the probability that is expected using the frequencies of the two words and assuming that their occurrence is independent ((Church and Gale, 1991), figure 5). Furthermore, using held out data …

Content truncated to 500 words