A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1599–1609, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution Shaohua Li1, Jun Zhu2, Chunyan Miao1 1Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), Nanyang Technological University, Singapore 2Tsinghua University, P.R. China lish0018@ntu.edu.sg, dcszj@tsinghua.edu.cn, ascymiao@ntu.edu.sg Abstract Most existing word embedding methods can be categorized into Neural Embedding Models and Matrix Factorization (MF)- based methods. However some mod- els are opaque to probabilistic interpre- tation, and MF-based methods, typically solved using Singular Value Decomposi- tion (SVD), may incur loss of corpus in- formation. In addition, it is desirable to incorporate global latent factors, such as topics, sentiments or writing styles, into the word embedding model. Since gen- erative models provide a principled way to incorporate latent factors, we propose a generative word embedding model, which is easy to interpret, and can serve as a basis of more sophisticated latent factor models. The model inference reduces to a low rank weighted positive semidefinite approximation problem. Its optimization is approached by eigendecomposition on a submatrix, followed by online blockwise regression, which is scalable and avoids the information loss in SVD. In experi- ments on 7 common benchmark datasets, our vectors are competitive to word2vec, and better than other MF-based methods. 1 Introduction The task of word embedding is to model the distri- bution of a word and its context words using their corresponding vectors in a Euclidean space. Then by doing regression on the relevant statistics de- rived from a corpus, a set of vectors are recovered which best fit these statistics. These vectors, com- monly referred to as the embeddings, capture se- mantic/syntactic regularities between the words. The core of a word embedding method is the link function that connects the input — the embed- dings, with the output — certain corpus statistics. Based on the link function, the objective function is developed. The reasonableness of the link func- tion impacts the quality of the obtained embed- dings, and different link functions are amenable to different optimization algorithms, with different scalability. Based on the forms of the link func- tion and the optimization techniques, most meth- ods can be divided into two classes: the traditional neural embedding models, and more recent low rank matrix factorization methods. The neural embedding models use the softmax link function to model the conditional distribution of a word given its context (or vice versa) as a function of the embeddings. The normalizer in the softmax function brings intricacy to the optimiza- tion, which is usually tackled by gradient-based methods. The pioneering work was (Bengio et al., 2003). Later Mnih and Hinton (2007) propose three different link functions. However there are interaction matrices between the embeddings in all these models, which complicate and slow down the training, hindering them from being trained on huge corpora. Mikolov et al. (2013a) and Mikolov et al. (2013b) greatly simplify the …

Content truncated to 500 words