D-GloVe: A Feasible Least Squares Model for Estimating Word Embedding Densities

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1849–1860, Osaka, Japan, December 11-17 2016. D-GloVe: A Feasible Least Squares Model for Estimating Word Embedding Densities∗ Shoaib Jameel and Steven Schockaert School of Computer Science and Informatics, Cardiff University. {JameelS1, S.Schockaert}@cardiff.ac.uk Abstract We propose a new word embedding model, inspired by GloVe, which is formulated as a feasible least squares optimization problem. In contrast to existing models, we explicitly represent the uncertainty about the exact definition of each word vector. To this end, we estimate the error that results from using noisy co-occurrence counts in the formulation of the model, and we model the imprecision that results from including uninformative context words. Our experimental results demonstrate that this model compares favourably with existing word embedding models. 1 Introduction Several vector space models for word meaning have already been proposed (Lund and Burgess, 1996; Landauer and Dumais, 1997; Turney and Pantel, 2010; Mikolov et al., 2013; Pennington et al., 2014). While there are considerable differences in how these vector space models are learned, most approaches represent words as vectors. However, a few authors have proposed models that represent words as regions or densities in a vector space (Erk, 2009; Vilnis and McCallum, 2015), motivated by the view that region or density based representations are better suited to model the diversity of word meaning, and can thus capture e.g. hyponymy in a natural way. In this paper, we also use densities in a low-dimensional vector space to represent word meaning. In contrast to previous work, however, we use densities for modelling our lack of knowledge about the precise meaning of a word. This allows us to use a more cautious representation for rare words, and leads to better confidence estimates in downstream tasks. Note that this use of densities is indeed fundamentally different from its use in previous work. For example, increasing the corpus size in our case will lead to more precise estimates (i.e. distributions with lower variance) while the models from (Erk, 2009; Vilnis and McCallum, 2015) may arrive at distributions with higher variance, reflecting the broader set of context windows that may be found. Our approach is based on the GloVe model for word embedding (Pennington et al., 2014). In particular, we also associate two vectors with each word i: the vector wi, which intuitively represents the meaning of word i, and the vector ˜wj, which intuitively represents how the occurrence of i in the context of another word j affects the meaning of that word. Moreover, we also use a least squares formulation to constrain these vectors such that wi · ˜wj reflects the co-occurrence statistics of words i and j. In contrast to GloVe, however, we explicitly model two factors that contribute to the residual error of this model: (i) the fact that corpus statistics for rare terms are not reliable and (ii) the fact that not all words are equally informative. This has two key advantages. First, it allows us to …

Content truncated to 500 words