Distributional Clustering of English Words

DISTRIBUTIONAL CLUSTERING OF ENGLISH WORDS Fernando Pereira AT&T Bell Laboratories 600 Mountain Ave. Murray Hill, NJ 07974, USA pereira@research, att. com Naftali Tishby Dept. of Computer Science Hebrew University Jerusalem 91904, Israel tishby@cs, hu]i. ac. il Lillian Lee Dept. of Computer Science Cornell University Ithaca, NY 14850, USA llee~cs, cornell, edu Abstract We describe and evaluate experimentally a method for clustering words according to their dis- tribution in particular syntactic contexts. Words are represented by the relative frequency distribu- tions of contexts in which they appear, and rela- tive entropy between those distributions is used as the similarity measure for clustering. Clusters are represented by average context distributions de- rived from the given words according to their prob- abilities of cluster membership. In many cases, the clusters can be thought of as encoding coarse sense distinctions. Deterministic annealing is used to find lowest distortion sets of clusters: as the an- nealing parameter increases, existing clusters be- come unstable and subdivide, yielding a hierarchi- cal "soft" clustering of the data. Clusters are used as the basis for class models of word coocurrence, and the models evaluated with respect to held-out test data. INTRODUCTION Methods for automatically classifying words ac- cording to their contexts of use have both scien- tific and practical interest. The scientific ques- tions arise in connection to distributional views of linguistic (particularly lexical) structure and also in relation to the question of lexical acqui- sition both from psychological and computational learning perspectives. From the practical point of view, word classification addresses questions of data sparseness and generalization in statistical language models, particularly models for deciding among alternative analyses proposed by a gram- mar. It is well known that a simple tabulation of fre- quencies of certain words participating in certain configurations, for example of frequencies of pairs of a transitive main verb and the head noun of its direct object, cannot be reliably used for compar- ing the likelihoods of different alternative configu- rations. The problemis that for large enough cor- pora the number of possible joint events is much larger than the number of event occurrences in the corpus, so many events are seen rarely or never, making their frequency counts unreliable estimates of their probabilities. Hindle (1990) proposed dealing with the sparseness problem by estimating the likelihood of unseen events from that of "similar" events that have been seen. For instance, one may estimate the likelihood of a particular direct object for a verb from the likelihoods of that direct object for similar verbs. This requires a reasonable defini- tion of verb similarity and a similarity estimation method. In Hindle's proposal, words are similar if we have strong statistical evidence that they tend to participate in the same events. His notion of similarity seems to agree with our intuitions in many cases, but it is not clear how it can be used directly to construct word classes and correspond- ing models of association. Our research addresses some of the same ques- tions and uses similar raw …

Content truncated to 500 words