Multiview LSA: Representation Learning via Generalized CCA

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 556–566, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Multiview LSA: Representation Learning via Generalized CCA Pushpendre Rastogi1 and Benjamin Van Durme1,2 and Raman Arora1 1Center for Language and Speech Processing 2Human Language Technology Center of Excellence Johns Hopkins University Abstract Multiview LSA (MVLSA) is a generalization of Latent Semantic Analysis (LSA) that sup- ports the fusion of arbitrary views of data and relies on Generalized Canonical Correlation Analysis (GCCA). We present an algorithm for fast approximate computation of GCCA, which when coupled with methods for han- dling missing values, is general enough to ap- proximate some recent algorithms for induc- ing vector representations of words. Exper- iments across a comprehensive collection of test-sets show our approach to be competitive with the state of the art. 1 Introduction Winograd (1972) wrote that: “Two sentences are paraphrases if they produce the same representation in the internal formalism for meaning”. This intu- ition is made soft in vector-space models (Turney and Pantel, 2010), where we say that expressions in language are paraphrases if their representations are close under some distance measure. One of the earliest linguistic vector space mod- els was Latent Semantic Analysis (LSA). LSA has been successfully used for Information Retrieval but it is limited in its reliance on a single matrix, or view, of term co-occurrences. Here we address the single-view limitation of LSA by demonstrating that the framework of Generalized Canonical Correla- tion Analysis (GCCA) can be used to perform Mul- tiview LSA (MVLSA). This approach allows for the use of an arbitrary number of views in the induc- tion process, including embeddings induced using other algorithms. We also present a fast approx- imate method for performing GCCA and approxi- mately recover the objective of (Pennington et al., 2014) while accounting for missing values. Our experiments show that MVLSA is com- petitive with state of the art approached for inducing vector representations of words and phrases. As a methodological aside, we discuss the (in-)significance of conclusions being drawn from comparisons done on small sized datasets. 2 Motivation LSA is an application of Principal Component Anal- ysis (PCA) to a term-document cooccurrence ma- trix. The principal directions found by PCA form the basis of the vector-space in which to represent the input terms (Landauer and Dumais, 1997). A drawback of PCA is that it can leverage only a sin- gle source of data and it is sensitive to scaling. An arguably better approach to representation learning is Canonical Correlation Analysis (CCA) that induces representations that are maximally cor- related across two views, allowing the utilization of two distinct sources of data. While an improvement over PCA, being limited to only two views is un- fortunate in light of the fact that many sources of data (perspectives) are frequently available in prac- tice. In such cases it is natural to extend CCA’s orig- inal objective of maximizing correlation between …

Content truncated to 500 words