Learning a Reversible Embedding Mapping using Bi-Directional Manifold Alignment

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3132–3139 August 1–6, 2021. ©2021 Association for Computational Linguistics 3132 Learning a Reversible Embedding Mapping using Bi-Directional Manifold Alignment Ashwinkumar Ganesan∗, Francis Ferraro, Tim Oates Dept. Of Computer Science & Electrical Engineering (CSEE), University Of Maryland Baltimore County (UMBC), MD, USA - 21250 gashwin1@umbc.edu, ferraro@umbc.edu, oates@cs.umbc.edu Abstract We propose a Bi-Directional Manifold Align- ment (BDMA) that learns a non-linear map- ping between two manifolds by explicitly training it to be bijective. We demonstrate BDMA by training a model for a pair of lan- guages rather than individual, directed source and target combinations, reducing the num- ber of models by 50%. We show that mod- els trained with BDMA in the “forward” (source to target) direction can successfully map words in the “reverse” (target to source) direction, yielding equivalent (or better) per- formance to standard unidirectional translation models where the source and target language is flipped. We also show how BDMA reduces the overall size of the model. 1 Introduction Learning continuous vector representations of em- beddings is an expensive exercise as it requires a large quantity of free text to train stable repre- sentations (Sahin et al., 2017). Learning word em- beddings in the English language is relatively easy since a model can make use of free text online from sources like Wikipedia, but it is challenging to learn embeddings for natural languages where the free text is limited (low-resource languages). Resource- constrained languages suffer from dual problems of reduced quality of embeddings and their vocab- ulary being small. Cross-lingual words embedding (CLWE) models alleviate this problem but are often linear mapping functions that align the source and target language manifolds, since non-linear map- ping functions such as neural networks are unidirec- tional and known to perform poorly as compared to their linear counterparts (Ruder et al., 2019). In this paper, we propose Bi-Directional Man- ifold Alignment (BDMA), which learns a re- versible, non-linear mapping function between two ∗This research was completed prior to joining Amazon. Figure 1: Mapping vector spaces with Bi-Directional Manifold Alignment (BDMA). f is the feedforward network. fa and fb are represent the forward and back- ward direction of flow through the network. In a shared BDMA network, the blue components represent net- work fully connected layers, orange are activation lay- ers during forward network flow while purple repre- sents activation layers in reverse flow. During reverse flow from output to input, the weight matrix is a trans- pose of weights during forward flow through the net- work. manifolds. Inspired by CycleGAN (Zhu et al., 2017), we use a cycle consistency loss to optimize BDMA. We study BDMA in the context of cross- lingual lexicon induction and show that it offers so- lutions to two known problems: (1) that non-linear models are known to perform poorly in comparison to their linear counterparts (Ruder et al., 2019), and (2) most approaches perform unidirectional map- ping only (from a source to target language), lead- ing to an ever …

Content truncated to 500 words