Polarity Inducing Latent Semantic Analysis

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1212–1222, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Polarity Inducing Latent Semantic Analysis Wen-tau Yih Geoffrey Zweig John C. Platt Microsoft Research One Microsoft Way Redmond, WA 98052, USA {scottyih,gzweig,jplatt}@microsoft.com Abstract Existing vector space models typically map synonyms and antonyms to similar word vec- tors, and thus fail to represent antonymy. We introduce a new vector space representation where antonyms lie on opposite sides of a sphere: in the word vector space, synonyms have cosine similarities close to one, while antonyms are close to minus one. We derive this representation with the aid of a thesaurus and latent semantic analysis (LSA). Each entry in the thesaurus – a word sense along with its synonyms and antonyms – is treated as a “document,” and the resulting doc- ument collection is subjected to LSA. The key contribution of this work is to show how to as- sign signs to the entries in the co-occurrence matrix on which LSA operates, so as to induce a subspace with the desired property. We evaluate this procedure with the Grad- uate Record Examination questions of (Mo- hammed et al., 2008) and find that the method improves on the results of that study. Further improvements result from refining the sub- space representation with discriminative train- ing, and augmenting the training data with general newspaper text. Altogether, we im- prove on the best previous results by 11 points absolute in F measure. 1 Introduction Vector space representations have proven useful across a wide variety of text processing applications ranging from document clustering to search rele- vance measurement. In these applications, text is represented as a vector in a multi-dimensional con- tinuous space, and a similarity metric such as co- sine similarity can be used to measure the related- ness of different items. Vector space representations have been used both at the document and word lev- els. At the document level, they are effective for applications including information retrieval (Salton and McGill, 1983; Deerwester et al., 1990), docu- ment clustering (Deerwester et al., 1990; Xu et al., 2003), search relevance measurement (Baeza-Yates and Ribiero-Neto, 1999) and cross-lingual docu- ment retrieval (Platt et al., 2010). At the word level, vector representations have been used to measure word similarity (Deerwester et al., 1990; Turney and Littman, 2005; Turney, 2006; Turney, 2001; Lin, 1998; Agirre et al., 2009; Reisinger and Mooney, 2010) and for language modeling (Bellegarda, 2000; Coccaro and Jurafsky, 1998). While quite success- ful, these applications have typically been consistent with a very general notion of similarity in which basic association is measured, and finer shades of meaning need not be distinguished. For example, latent semantic analysis might assign a high degree of similarity to opposites as well as synonyms (Lan- dauer and Laham, 1998; Landauer, 2002). Independent of vector-space representations, a number of authors have focused on identifying dif- ferent kinds of relatedness. At the simplest …

Content truncated to 500 words