Probabilistic Modeling of Joint-context in Distributional Similarity

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 181–190, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Probabilistic Modeling of Joint-context in Distributional Similarity Oren Melamud§, Ido Dagan§, Jacob Goldberger♦, Idan Szpektor†, Deniz Yuret‡ § Computer Science Department, Bar-Ilan University ♦ Faculty of Engineering, Bar-Ilan University † Yahoo! Research Israel ‡ Koc¸ University {melamuo,dagan,goldbej}@{cs,cs,eng}.biu.ac.il idan@yahoo-inc.com, dyuret@ku.edu.tr Abstract Most traditional distributional similarity models fail to capture syntagmatic patterns that group together multiple word features within the same joint context. In this work we introduce a novel generic distributional similarity scheme under which the power of probabilistic models can be leveraged to effectively model joint contexts. Based on this scheme, we implement a concrete model which utilizes probabilistic n-gram language models. Our evaluations sug- gest that this model is particularly well- suited for measuring similarity for verbs, which are known to exhibit richer syntag- matic patterns, while maintaining compa- rable or better performance with respect to competitive baselines for nouns. Fol- lowing this, we propose our scheme as a framework for future semantic similarity models leveraging the substantial body of work that exists in probabilistic language modeling. 1 Introduction The Distributional Hypothesis is commonly phrased as “words which are similar in meaning occur in similar contexts” (Rubenstein and Good- enough, 1965). Distributional similarity models following this hypothesis vary in two major as- pects, namely the representation of the context and the respective computational model. Probably the most prominent class of distributional similarity models represents context as a vector of word fea- tures and computes similarity using feature vector arithmetics (Lund and Burgess, 1996; Turney et al., 2010). To construct the feature vectors, the context of each target word token1, which is com- monly a word window around it, is first broken 1We use word type to denote an entry in the vocabulary, and word token for a particular occurrence of a word type. into a set of individual independent words. Then the weights of the entries in the word feature vec- tor capture the degree of association between the target word type and each of the individual word features, independently of one another. Despite its popularity, it was suggested that the word feature vector approach misses valu- able information, which is embedded in the co- location and inter-relations of words (e.g. word order) within the same context (Ruiz-Casado et al., 2005). Following this motivation, Ruiz-Casado et al. (2005) proposed an alternative composite- feature model, later adopted in (Agirre et al., 2009). This model adopts a richer context repre- sentation by considering entire word window con- texts as features, while keeping the same compu- tational vector-based model. Although showing interesting potential, this approach suffers from a very high-dimensional feature space resulting in data sparseness problems. Therefore, it requires exceptionally large learning corpora to consider large windows effectively. A parallel line of work adopted richer context representations as well, with a different compu- tational model. These works utilized neural net- works to learn low dimensional continuous vector representations for …

Content truncated to 500 words