Density Matching for Bilingual Word Embedding

Proceedings of NAACL-HLT 2019, pages 1588–1598 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1588 Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma, Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University ctzhou,xuezhem,diwang,gneubig@cs.cmu.edu Abstract Recent approaches to cross-lingual word em- bedding have generally been based on lin- ear transformations between the sets of em- bedding vectors in the two languages. In this paper, we propose an approach that in- stead expresses the two monolingual embed- ding spaces as probability densities defined by a Gaussian mixture model, and matches the two densities using a method called normal- izing flow. The method requires no explicit supervision, and can be learned with only a seed dictionary of words that have identical strings. We argue that this formulation has several intuitively attractive properties, partic- ularly with the respect to improving robust- ness and generalization to mappings between difficult language pairs or word pairs. On a benchmark data set of bilingual lexicon in- duction and cross-lingual word similarity, our approach can achieve competitive or superior performance compared to state-of-the-art pub- lished results, with particularly strong results being found on etymologically distant and/or morphologically rich languages.1 1 Introduction Cross-lingual word embeddings represent words in different languages in a single vector space, capturing the syntactic and semantic similarity of words across languages in a way conducive to use in computational models (Upadhyay et al., 2016; Ruder et al., 2017). These embeddings have been shown to be an effective tool for cross-lingual NLP, e.g. the transfer of models trained on high- resource languages to low-resource ones (Klemen- tiev et al., 2012; Guo et al., 2015; Zoph et al., 2016; Zhang et al., 2018; Gu et al., 2018) or un- supervised learning (Artetxe et al., 2018c). 1Code/scripts can be found at https://github. com/violet-zct/DeMa-BWE. dog cat canine ⽝犬 猫 ⿃鳥 mapping function bird Japanese Space English Space Figure 1: An illustration of our method. Thicker lines represent higher mixture weights, linked to word fre- quency. The pink point is a continuous training sample in the Japanese embedding space while the blue point is a mapped point in the English space. There are two major paradigms in the learn- ing of cross-lingual word embeddings: “online” and “offline”. “Online” methods learn the cross- lingual embeddings directly from parallel corpora (Hermann and Blunsom, 2014), optionally aug- mented with monolingual corpora (Gouws et al., 2015). In contrast, “offline” approaches learn a bilingual mapping function or multilingual pro- jections from pre-trained monolingual word em- beddings or feature vectors (Haghighi et al., 2008; Mikolov et al., 2013; Faruqui and Dyer, 2014). In this work, we focus on this latter offline approach. The goal of bilingual embedding is to learn a shared embedding space where words possessing similar meanings are projected to nearby points. Early work focused on supervised methods maxi- mizes the similarity of the embeddings of words that exist in a manually-created dictionary, ac- cording to some similarity metric (Mikolov et al., 2013; Faruqui and Dyer, 2014; …

Content truncated to 500 words