Improving Bilingual Projections via Sparse Covariance Matrices

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 930–940, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Improving Bilingual Projections via Sparse Covariance Matrices Jagadeesh Jagarlamudi University of Maryland College Park, USA jags@umiacs.umd.edu Raghavendra Udupa Microsoft Research Bangalore, India raghavu@microsoft.com Hal Daum´e III University of Maryland College Park, USA hal@umiacs.umd.edu Abhijit Bhole Microsoft Research Bangalore, India v-abbhol@microsoft.com Abstract Mapping documents into an interlingual rep- resentation can help bridge the language bar- rier of cross-lingual corpora. Many existing approaches are based on word co-occurrences extracted from aligned training data, repre- sented as a covariance matrix. In theory, such a covariance matrix should represent seman- tic equivalence, and should be highly sparse. Unfortunately, the presence of noise leads to dense covariance matrices which in turn leads to suboptimal document representations. In this paper, we explore techniques to recover the desired sparsity in covariance matrices in two ways. First, we explore word association measures and bilingual dictionaries to weigh the word pairs. Later, we explore different selection strategies to remove the noisy pairs based on the association scores. Our experi- mental results on the task of aligning compa- rable documents shows the efficacy of sparse covariance matrices on two data sets from two different language pairs. 1 Introduction Aligning documents from different languages arises in a range of tasks such as parallel phrase extrac- tion (Gale and Church, 1991; Rapp, 1999), mining translations for out-of-vocabulary words for statis- tical machine translation (Daume III and Jagarla- mudi, 2011) and document retrieval (Ballesteros and Croft, 1996; Munteanu and Marcu, 2005). In this task, we are given a comparable corpora and some documents in one language are assumed to have a comparable document in the other language and the goal is to recover this hidden alignment. In this pa- per, we address this problem by mapping the docu- ments into a common subspace (interlingual repre- sentation). This common subspace generalizes the notion of vector space model for cross-lingual ap- plications (Turney and Pantel, 2010). Most of the existing approaches use manually aligned document pairs to find a common subspace in which the aligned document pairs are maximally correlated. The sub-space can be found using ei- ther generative approaches based on topic modeling (Mimno et al., 2009; Jagarlamudi and Daum´e III, 2010; Zhang et al., 2010; Vu et al., 2009) or dis- criminative approaches based on variants of Princi- pal Component Analysis (PCA) and Canonical Cor- relation Analysis (CCA) (Susan T. Dumais, 1996; Vinokourov et al., 2003; Platt et al., 2010; Haghighi et al., 2008). Both styles rely on document level term co-occurrences to find the latent representation. The discriminative approaches capture essential word co-occurrences in terms of two monolingual covariance matrices and a cross-covariance matrix. Subsequently, they use these covariance matrices to find projection directions in each language such that aligned documents lie close to each other (Sec. 2). The strong reliance of these approaches on the co- variance matrices leads to problems, especially with the noisy data …

Content truncated to 500 words