Learning principled bilingual mappings of word embeddings while preserving monolingual invariance

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Learning principled bilingual mappings of word embeddings while preserving monolingual invariance Mikel Artetxe, Gorka Labaka, Eneko Agirre IXA NLP Group, University of the Basque Country (UPV/EHU) {mikel.artetxe, gorka.labaka, e.agirre}@ehu.eus Abstract Mapping word embeddings of different lan- guages into a single space has multiple appli- cations. In order to map from a source space into a target space, a common approach is to learn a linear mapping that minimizes the dis- tances between equivalences listed in a bilin- gual dictionary. In this paper, we propose a framework that generalizes previous work, provides an efficient exact method to learn the optimal linear transformation and yields the best bilingual results in translation induction while preserving monolingual performance in an analogy task. 1 Introduction Bilingual word embeddings have attracted a lot of attention in recent times (Zou et al., 2013; Koˇcisk´y et al., 2014; Chandar A P et al., 2014; Gouws et al., 2014; Gouws and Søgaard, 2015; Luong et al., 2015; Wick et al., 2016). A common approach to obtain them is to train the embeddings in both languages independently and then learn a mapping that mini- mizes the distances between equivalences listed in a bilingual dictionary. The learned transformation can also be applied to words missing in the dictionary, which can be used to induce new translations with a direct application in machine translation (Mikolov et al., 2013b; Zhao et al., 2015). The first method to learn bilingual word em- bedding mappings was proposed by Mikolov et al. (2013b), who learn the linear transformation that minimizes the sum of squared Euclidean distances for the dictionary entries. Subsequent work has pro- posed alternative optimization objectives to learn better mappings. Xing et al. (2015) incorporate length normalization in the training of word embed- dings and try to maximize the cosine similarity in- stead, introducing an orthogonality constraint to pre- serve the length normalization after the projection. Faruqui and Dyer (2014) use canonical correlation analysis to project the embeddings in both languages to a shared vector space. Beyond linear mappings, Lu et al. (2015) apply deep canonical correlation analysis to learn a non- linear transformation for each language. Finally, ad- ditional techniques have been used to address the hubness problem in Mikolov et al. (2013b), both through the neighbor retrieval method (Dinu et al., 2015) and the training itself (Lazaridou et al., 2015). We leave the study of non-linear transformation and other additions for further work. In this paper, we propose a general framework to learn bilingual word embeddings. We start with a basic optimization objective (Mikolov et al., 2013b) and introduce several meaningful and intuitive con- straints that are equivalent or closely related to pre- viously proposed methods (Faruqui and Dyer, 2014; Xing et al., 2015). Our framework provides a more general view of bilingual word embedding map- pings, showing the underlying connection between the existing methods, revealing some flaws …

Content truncated to 500 words