Model-based Word Embeddings from Decompositions of Count Matrices

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1282–1291, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Model-based Word Embeddings from Decompositions of Count Matrices Karl Stratos Michael Collins Daniel Hsu Columbia University, New York, NY 10027, USA {stratos, mcollins, djhsu}@cs.columbia.edu Abstract This work develops a new statistical un- derstanding of word embeddings induced from transformed count data. Using the class of hidden Markov models (HMMs) underlying Brown clustering as a genera- tive model, we demonstrate how canoni- cal correlation analysis (CCA) and certain count transformations permit efficient and effective recovery of model parameters with lexical semantics. We further show in experiments that these techniques empir- ically outperform existing spectral meth- ods on word similarity and analogy tasks, and are also competitive with other pop- ular methods such as WORD2VEC and GLOVE. 1 Introduction The recent spike of interest in dense, low- dimensional lexical representations—i.e., word embeddings—is largely due to their ability to cap- ture subtle syntactic and semantic patterns that are useful in a variety of natural language tasks. A successful method for deriving such embed- dings is the negative sampling training of the skip-gram model suggested by Mikolov et al. (2013b) and implemented in the popular software WORD2VEC. The form of its training objective was motivated by efficiency considerations, but has subsequently been interpreted by Levy and Goldberg (2014b) as seeking a low-rank factor- ization of a matrix whose entries are word-context co-occurrence counts, scaled and transformed in a certain way. This observation sheds new light on WORD2VEC, yet also raises several new ques- tions about word embeddings based on decompos- ing count data. What is the right matrix to de- compose? Are there rigorous justifications for the choice of matrix and count transformations? In this paper, we answer some of these ques- tions by investigating the decomposition specified by CCA (Hotelling, 1936), a powerful technique for inducing generic representations whose com- putation is efficiently and exactly reduced to that of a matrix singular value decomposition (SVD). We build on and strengthen the work of Stratos et al. (2014) which uses CCA for learning the class of HMMs underlying Brown clustering. We show that certain count transformations enhance the ac- curacy of the estimation method and significantly improve the empirical performance of word rep- resentations derived from these model parameters (Table 1). In addition to providing a rigorous justifica- tion for CCA-based word embeddings, we also supply a general template that encompasses a range of spectral methods (algorithms employing SVD) for inducing word embeddings in the lit- erature, including the method of Levy and Gold- berg (2014b). In experiments, we demonstrate that CCA combined with the square-root transforma- tion achieves the best result among spectral meth- ods and performs competitively with other popu- lar methods such as WORD2VEC and GLOVE on word similarity and analogy tasks. We addition- ally demonstrate that CCA embeddings provide the most competitive improvement when used as …

Content truncated to 500 words