Multi-Relational Latent Semantic Analysis

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1602–1612, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Multi-Relational Latent Semantic Analysis Kai-Wei Chang∗ University of Illinois Urbana, IL 61801, USA kchang10@illinois.edu Wen-tau Yih Christopher Meek Microsoft Research Redmond, WA 98052, USA {scottyih,meek}@microsoft.com Abstract We present Multi-Relational Latent Seman- tic Analysis (MRLSA) which generalizes La- tent Semantic Analysis (LSA). MRLSA pro- vides an elegant approach to combining mul- tiple relations between words by construct- ing a 3-way tensor. Similar to LSA, a low- rank approximation of the tensor is derived using a tensor decomposition. Each word in the vocabulary is thus represented by a vec- tor in the latent semantic space and each re- lation is captured by a latent square matrix. The degree of two words having a specific relation can then be measured through sim- ple linear algebraic operations. We demon- strate that by integrating multiple relations from both homogeneous and heterogeneous information sources, MRLSA achieves state- of-the-art performance on existing benchmark datasets for two relations, antonymy and is-a. 1 Introduction Continuous semantic space representations have proven successful in a wide variety of NLP and IR applications, such as document clustering (Xu et al., 2003) and cross-lingual document retrieval (Dumais et al., 1997; Platt et al., 2010) at the document level and sentential semantics (Guo and Diab, 2012; Guo and Diab, 2013) and syntactic parsing (Socher et al., 2013) at the sentence level. Such representa- tions also play an important role in applications for lexical semantics, such as word sense disambigua- tion (Boyd-Graber et al., 2007), measuring word ∗Work conducted while interning at Microsoft Research. similarity (Deerwester et al., 1990) and relational similarity (Turney, 2006; Zhila et al., 2013; Mikolov et al., 2013). In many of these applications, La- tent Semantic Analysis (LSA) (Deerwester et al., 1990) has been widely used, serving as a fundamen- tal component or as a strong baseline. LSA operates by mapping text objects, typically documents and words, to a latent semantic space. The proximity of the vectors in this space implies that the original text objects are semantically re- lated. However, one well-known limitation of LSA is that it is unable to differentiate fine-grained re- lations. For instance, when applied to lexical se- mantics, synonyms and antonyms may both be as- signed high similarity scores (Landauer and Laham, 1998; Landauer, 2002). Asymmetric relations like hyponyms and hypernyms also cannot be differenti- ated. Although there exists some recent work, such as PILSA which tries to overcome this weakness of LSA by introducing the notion of polarity (Yih et al., 2012). This extension, however, can only handle two opposing relations (e.g., synonyms and antonyms), leaving open the challenge of encoding multiple relations. In this paper, we propose Multi-Relational Latent Semantic Analysis (MRLSA), which strictly gener- alizes LSA to incorporate information of multiple relations concurrently. Similar to LSA or PILSA when applied to lexical semantics, each word is still mapped to a vector in the latent space. However, when …

Content truncated to 500 words