Word and Document Embedding with vMF-Mixture Priors on Context Word Vectors

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3319–3328 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3319 Word and Document Embedding with vMF-Mixture Priors on Context Word Vectors Shoaib Jameel Medway School of Computing University of Kent M.S.Jameel@kent.ac.uk Steven Schockaert School of Computer Science and Informatics Cardiff University SchockaertS1@cardiff.ac.uk Abstract Word embedding models typically learn two types of vectors: target word vectors and con- text word vectors. These vectors are normally learned such that they are predictive of some word co-occurrence statistic, but they are oth- erwise unconstrained. However, the words from a given language can be organized in various natural groupings, such as syntactic word classes (e.g. nouns, adjectives, verbs) and semantic themes (e.g. sports, politics, sen- timent). Our hypothesis in this paper is that embedding models can be improved by explic- itly imposing a cluster structure on the set of context word vectors. To this end, our model relies on the assumption that context word vec- tors are drawn from a mixture of von Mises- Fisher (vMF) distributions, where the parame- ters of this mixture distribution are jointly op- timized with the word vectors. We show that this results in word vectors which are qualita- tively different from those obtained with exist- ing word embedding models. We furthermore show that our embedding model can also be used to learn high-quality document represen- tations. 1 Introduction Word embedding models are aimed at learning vector representations of word meaning (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017). These representations are primarily learned from co-occurrence statistics, where two words are represented by similar vectors if they tend to occur in similar linguistic contexts. Most models, such as Skip-gram (Mikolov et al., 2013b) and GloVe (Pennington et al., 2014) learn two dif- ferent vector representations w and ˜w for each word w, which we will refer to as the target word vector and the context word vector respectively. Apart from the constraint that wi · ˜ wj should re- flect how often words wi and wj co-occur, these vectors are typically unconstrained. As was shown in (Mu et al., 2018), after per- forming a particular linear transformation, the an- gular distribution of the word vectors that are ob- tained by standard models is essentially uniform. This isotropy property is convenient for study- ing word embeddings from a theoretical point of view (Arora et al., 2016), but it sits at odds with fact that words can be organised in various nat- ural groupings. For instance, we might perhaps expect that words from the same part-of-speech class should be clustered together in the word em- bedding. Similarly, we might expect that organ- ising word vectors in clusters that represent se- mantic themes would also be beneficial. In fact, a number of approaches have already been proposed that use external knowledge for imposing such a cluster structure, capturing the intuition that words which belong to the same category should be …

Content truncated to 500 words