Quantized Wasserstein Procrustes Alignment of Word Embedding Spaces

Quantized Wasserstein Procrustes Alignment of Word Embedding Spaces Prince Osei Aboagye1∗, Yan Zheng2, Chin-Chia Michael Yeh2, Junpeng Wang2, Zhongfang Zhuang2, Huiyuan Chen2, Liang Wang2, Wei Zhang2, Jeff M. Phillips1 1University of Utah, 2Visa Research 1{prince ,jeffp}@cs.utah.edu 2{yazheng ,miyeh ,junpenwa ,zzhuang ,hchen ,liawang,wzhan}@visa.com Abstract Optimal Transport (OT) provides a useful geometric framework to estimate the permutation matrix under unsupervised cross-lingual word embedding (CLWE) models that pose the align- ment task as a Wasserstein-Procrustes problem. However, linear programming algorithms and approximate OT solvers via Sinkhorn for computing the permutation matrix come with a significant computational burden since they scale cubically and quadratically, respectively, in the input size. This makes it slow and infeasible to compute OT distances exactly for a larger input size, resulting in a poor approximation quality of the permutation matrix and subse- quently a less robust learned transfer function or mapper. This paper proposes an unsupervised projection-based CLWE model called quantized Wasserstein Procrustes (qWP). qWP relies on a quantization step of both the source and target monolingual embedding space to estimate the permutation matrix given a cheap sampling procedure. This approach substantially improves the approximation quality of empirical OT solvers given fixed computational cost. We demonstrate that qWP achieves state-of-the-art results on the Bilingual lexicon Induction (BLI) task. 1 Introduction In natural language processing (NLP), the problem of aligning monolingual embedding spaces to induce a shared cross-lingual vector space has been shown not only to be useful in a variety of tasks such as bilingual lexicon induction (BLI) (Mikolov et al., 2013; Barone, 2016; Artetxe et al., 2017; Aboagye et al., 2022), machine translation (Artetxe et al., 2018b), cross-lingual information retrieval (Vuli´c & Moens, 2015), but it plays a crucial role in facilitating the cross-lingual transfer of language technologies from high resource languages to low resource languages. Cross-lingual word embeddings (CLWEs) represent words from two or more languages in a shared cross-lingual vector space in which words with similar meanings obtain similar vectors regardless of their language. There has been a flurry of work dominated by the so-called projection-based CLWE models (Mikolov et al., 2013; Artetxe et al., 2016, 2017, 2018a; Smith et al., 2017; Ruder et al., 2019), which aim to improve CLWE model performance significantly. Projection-based CLWE models learn a transfer function or mapper between two independently trained monolingual word vector spaces with limited or no cross-lingual supervision. Famous among projection-based CLWE models are the unsupervised projection-based CLWE models (Artetxe et al., 2017; Lample et al., 2018; Alvarez-Melis & Jaakkola, 2018; ∗work done while interning at Visa Research Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 200 Grave et al., 2019): they eliminate the initial seed bilingual lexicon and rely on the topological similarities between monolingual spaces, known as the isometry assumption, to extract seed bilingual lexicons. This makes them attractive since they require no cross-lingual supervision. One of the ways of framing unsupervised CLWE models is to pose the alignment …

Content truncated to 500 words