Deep Multilingual Correlation for Improved Word Embeddings

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 250–256, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Deep Multilingual Correlation for Improved Word Embeddings Ang Lu1, Weiran Wang2, Mohit Bansal2, Kevin Gimpel2, and Karen Livescu2 1Department of Automation, Tsinghua University, Beijing, 100084, China lva11@mails.tsinghua.edu.cn 2Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {weiranwang, mbansal, kgimpel, klivescu}@ttic.edu Abstract Word embeddings have been found useful for many NLP tasks, including part-of-speech tagging, named entity recognition, and pars- ing. Adding multilingual context when learn- ing embeddings can improve their quality, for example via canonical correlation analysis (CCA) on embeddings from two languages. In this paper, we extend this idea to learn deep non-linear transformations of word embed- dings of the two languages, using the recently proposed deep canonical correlation analy- sis. The resulting embeddings, when eval- uated on multiple word and bigram similar- ity tasks, consistently improve over monolin- gual embeddings and over embeddings trans- formed with linear CCA. 1 Introduction Learned word representations are widely used in NLP tasks such as tagging, named entity recogni- tion, and parsing (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010; T¨ackstr¨om et al., 2012; Huang et al., 2014; Bansal et al., 2014). The idea in such representations is that words with similar context have similar meaning, and hence should be nearby in a clustering or vector space. Con- tinuous representations are learned with neural lan- guage models (Bengio et al., 2003; Mnih and Hin- ton, 2007; Mikolov et al., 2013) or spectral meth- ods (Deerwester et al., 1990; Dhillon et al., 2011). The context used to learn these representations is typically the set of nearby words of each word oc- currence. Prior work has found that adding transla- tional context results in better representations (Diab and Resnik, 2002; T¨ackstr¨om et al., 2012; Bansal et al., 2012; Zou et al., 2013). Recently, Faruqui and Dyer (2014) applied canonical correlation analysis (CCA) to word embeddings of two languages, and found that the resulting embeddings represent word similarities better than the original monolingual em- beddings. In this paper, we follow the same intuition as Faruqui and Dyer (2014) but rather than learning lin- ear transformations with CCA, we permit the cor- related information to lie in nonlinear subspaces of the original embeddings. We use the recently pro- posed deep canonical correlation analysis (DCCA) technique of Andrew et al. (2013) to learn non- linear transformations of two languages’ embed- dings that are highly correlated. We evaluate our DCCA-transformed embeddings on word similarity tasks like WordSim-353 (Finkelstein et al., 2001) and SimLex-999 (Hill et al., 2014), and also on the bigram similarity task of Mitchell and Lapata (2010) (using additive composition), obtaining con- sistent improvements over the original embeddings and over linear CCA. We also compare tuning crite- ria and ensemble methods for these architectures. 2 Method We assume that we have initial word embeddings for two languages, denoted by random vectors x …

Content truncated to 500 words