Low-Dimensional Manifold Distributional Semantic Models

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 731–740, Dublin, Ireland, August 23-29 2014. Low-Dimensional Manifold Distributional Semantic Models Georgia Athanasopoulou School of Electronic & Computer Engineering T.U.C. Chania, Greece gathanasopoulou@isc.tuc.gr Elias Iosif Athena Research and Innovation Center, 15125 Maroussi, Greece iosif.elias@gmail.com Alexandros Potamianos School of Electrical & Computer Engineering N.T.U.A, Athens, Greece apotam@gmail.com Abstract Motivated by evidence in psycholinguistics and cognition, we propose a hierarchical distributed semantic model (DSM) that consists of low-dimensional manifolds built on semantic neighbor- hoods. Each semantic neighborhood is sparsely encoded and mapped into a low-dimensional space. Global operations are decomposed into local operations in multiple sub-spaces; results from these local operations are fused to come up with semantic relatedness estimates. Manifold DSM are constructed starting from a pairwise word-level semantic similarity matrix. The pro- posed model is evaluated on semantic similarity estimation task significantly improving on the state-of-the-art. 1 Introduction The estimation of semantic similarity between words, sentences and documents is a fundamental problem for many research disciplines including computational linguistics (Malandrakis et al., 2011), semantic web (Corby et al., 2006), cognitive science and artificial intelligence (Resnik, 2011; Budanitsky and Hirst, 2001). In this paper, we study the geometrical structure of the lexical space in order to extract se- mantic relations among words. In (Karlgren et al., 2008), the high-dimensional lexical space is assumed to consist of manifolds of very low dimensionality that are embedded in this high dimensional space. The manifold hypothesis is compatible with evidence from psycholinguistics and cognitive science. In (Tenenbaum et al., 2011), the question “How does the mind work?” is answered as follows: cognitive organization is based on domains with similar items connected to each other and lexical information is represented hierarchically, i.e., a domain that consists of similar lexical entries may be represented by a more abstract concept. An example of such a domain is {blue, red, yellow, pink, ...} that corre- sponds by the concept of color. An inspiring analysis about the geometry of thought, as well as cognitive evidence for the low-dimensional manifold assumption can be found in (Gardenfors, 2000), e.g., the domain of color is argued to be cognitively represented as an one-dimensional manifold. Following the low-dimensional manifold hypothesis we propose to extend distributional semantic models (DSMs) into a hierarchical model of domains (or concepts) that contain semantically similar words. Global operations on the lexical space are decomposed into local operations on the low-dimensional domain sub-manifolds. Our goal is to exploit this hierarchical low-rank model to estimate relations between words, such as se- mantic similarity. There has been much research interest on devising data-driven approaches for estimating semantic similarity between words. DSMs (Baroni and Lenci, 2010) are based on the distributional hypothesis of meaning (Harris, 1954) assuming that semantic similarity between words is a function of the overlap of their linguistic contexts. DSMs are typically constructed from co-occurrence statistics of word tuples that are extracted on existing corpora or on corpora specifically harvested from the web. In (Iosif and …

Content truncated to 500 words