Embedding Words as Distributions with a Bayesian Skip-gram Model

Proceedings of the 27th International Conference on Computational Linguistics, pages 1775–1789 Santa Fe, New Mexico, USA, August 20-26, 2018. 1775 Embedding Words as Distributions with a Bayesian Skip-gram Model Arthur Braˇzinskas1 Serhii Havrylov2 Ivan Titov1,2 1ILLC, University of Amsterdam 2ILCC, School of Informatics, University of Edinburgh arthur.brazinskas@gmail.com s.havrylov@ed.ac.uk ititov@inf.ed.ac.uk Abstract We introduce a method for embedding words as probability densities in a low-dimensional space. Rather than assuming that a word embedding is fixed across the entire text collection, as in stan- dard word embedding methods, in our Bayesian model we generate it from a word-specific prior density for each occurrence of a given word. Intuitively, for each word, the prior density en- codes the distribution of its potential ‘meanings’. These prior densities are conceptually similar to Gaussian embeddings of Vilnis and McCallum (2015). Interestingly, unlike the Gaussian em- beddings, we can also obtain context-specific densities: they encode uncertainty about the sense of a word given its context and correspond to the approximate posterior distributions within our model. The context-dependent densities have many potential applications: for example, we show that they can be directly used in the lexical substitution task. We describe an effective estimation method based on the variational autoencoding framework and demonstrate the effectiveness of our embedding technique on a range of standard benchmarks. 1 Introduction Distributed representations of words induced from large unlabeled text collections have had a large im- pact on many natural language processing (NLP) applications, providing an effective and simple way of dealing with data sparsity. Word embedding methods typically represent words as vectors in a low- dimensional space (Deerwester et al., 1990; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014). In contrast, we encode them as probability densities (see Figure 1). Intuitively, the densities represent the distributions over possible meanings1 of a word. Representing a word as a distribution provides many potential benefits. For example, such embeddings let us encode generality of terms (e.g., kakapo is a hypernym of bird), characterize uncertainty about semantic properties of the correspond- ing referent (e.g., a proper noun, such as John, encodes little about the person it refers to) or represent polysemy (e.g., kiwi may refer to a fruit, a bird or a New Zealander). The main inspiration for this work is Gaussian embeddings (word2gauss, W2G) introduced by Vilnis and McCallum (2015). They represent words as Gaussian distributions and directly optimize an objec- tive expressed in terms of divergences (e.g., the Kullback-Liebler divergence) between the distributions. In contrast, we approach the problem from the generative Bayesian perspective. Though, as in W2G, context-agnostic densities are present in our model (they correspond to data-dependent priors), unlike W2G, we can also perform posterior inference and obtain context-specific densities. These posterior densities encode semantic properties of a word in a given context. For example, in Figure 1, when ‘kiwi‘ appears in a context suggesting the ‘bird’ sense, the posterior (represented by the shaded ellipsoid) be- comes more ‘peaky’ and moves towards the representation of word ‘bird’. We use …

Content truncated to 500 words