Factorization of Latent Variables in Distributional Semantic Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 227–231, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Factorization of Latent Variables in Distributional Semantic Models Arvid ¨Osterlund and David ¨Odling KTH Royal Institute of Technology, Sweden arvidos|dodling@kth.se Magnus Sahlgren Gavagai, Sweden mange@gavagai.se Abstract This paper discusses the use of factoriza- tion techniques in distributional semantic models. We focus on a method for re- distributing the weight of latent variables, which has previously been shown to im- prove the performance of distributional se- mantic models. However, this result has not been replicated and remains poorly un- derstood. We refine the method, and pro- vide additional theoretical justification, as well as empirical results that demonstrate the viability of the proposed approach. 1 Introduction Distributional Semantic Models (DSMs) have be- come standard paraphernalia in the natural lan- guage processing toolbox, and even though there is a wide variety of models available, the basic parameters of DSMs (context type and size, fre- quency weighting, and dimension reduction) are now well understood. This is demonstrated by the recent convergence of state-of-the-art results (Ba- roni et al., 2014; Levy and Goldberg, 2014). However, there are a few notable exceptions. One is the performance improvements demon- strated in two different papers using a method for redistributing the weight of principal components (PCs) in factorized DSMs (Caron, 2001; Bulli- naria and Levy, 2012). In the latter of these pa- pers, the factorization of latent variables in DSMs is used to reach a perfect score of 100% correct answers on the TOEFL synonym test. This re- sult is somewhat surprising, since the factorization method is the inverse of what is normally used. Neither the result nor the method has been repli- cated, and therefore remains poorly understood. The goal of this paper is to replicate and explain the result. In the following sections, we first pro- vide a brief review of DSMs and factorization, and review the method for redistributing the weight of latent variables. We then replicate the 100% score on the TOEFL test and provide additional state-of- the-art scores for the BLESS test. We also provide a more principled reformulation of the factoriza- tion method that is better suited for practical ap- plications. 2 Distributional Semantics Consider a set of words W = {w1, . . . , wn} and a set of context words C = {c1, . . . , cm}. The DSM representation is created by registering an occur- rence of a word wi with a set of context words cj, . . . , ck with a corresponding increment of the projection of wi on the cj, . . . , ck bases. In other words, each cell fij in the matrix representation F represents a co-occurrence count between a word wi and a context cj. In the following, we use W = C, making the co-occurrence matrix sym- metric Fn×n. We also adhere to standard practice of weighting the co-occurrence counts with Posi- tive Pointwise …

Content truncated to 500 words