LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2712–2723, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2712 LNMAP: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space Tasnim Mohiuddin¶, M Saiful Bari¶, and Shafiq Joty¶† ¶Nanyang Technological University, Singapore †Salesforce Research {mohi0004, bari0001, srjoty}@ntu.edu.sg Abstract Most of the successful and predominant meth- ods for Bilingual Lexicon Induction (BLI) are mapping-based, where a linear mapping func- tion is learned with the assumption that the word embedding spaces of different languages exhibit similar geometric structures (i.e., ap- proximately isomorphic). However, several re- cent studies have criticized this simplified as- sumption showing that it does not hold in gen- eral even for closely related languages. In this work, we propose a novel semi-supervised method to learn cross-lingual word embed- dings for BLI. Our model is independent of the isomorphic assumption and uses non-linear mapping in the latent space of two indepen- dently pre-trained autoencoders. Through ex- tensive experiments on fifteen (15) different language pairs (in both directions) compris- ing resource-rich and low-resource languages from two different datasets, we demonstrate that our method outperforms existing models by a good margin. Ablation studies show the importance of different model components and the necessity of non-linear mapping. 1 Introduction In recent years, a plethora of methods have been proposed to learn cross-lingual word embeddings (or CLWE for short) from monolingual word em- beddings. Here words with similar meanings in dif- ferent languages are represented by similar vectors, regardless of their actual language. CLWE enable us to compare the meaning of words across lan- guages, which is key to most multi-lingual applica- tions such as bilingual lexicon induction (Heyman et al., 2017), machine translation (Lample et al., 2018; Artetxe et al., 2018c), or multi-lingual infor- mation retrieval (Vuli´c and Moens, 2015). They also play a crucial role in cross-lingual knowledge transfer between languages (e.g., from resource- rich to low-resource languages) by providing a common representation space (Ruder et al., 2019). Mikolov et al. (2013a), in their pioneering work, learn a linear mapping function to transform the source embedding space to the target language by minimizing the squared Euclidean distance be- tween the translation pairs of a seed dictionary. They assume that the similarity of geometric ar- rangements in the embedding spaces is the key rea- son for their method to succeed as they found lin- ear mapping superior to non-linear mappings with multi-layer neural networks. Subsequent studies propose to improve the model by normalizing the embeddings, imposing an orthogonality constraint on the linear mapper, modifying the objective func- tion, and reducing the seed dictionary size (Artetxe et al., 2016, 2017, 2018a; Smith et al., 2017). A more recent line of research attempts to elimi- nate the seed dictionary totally and learn the map- ping in a purely unsupervised way (Barone, 2016; Zhang et al., 2017; Conneau et al., 2018; Artetxe et al., 2018b; Xu et al., 2018; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018; Mohiud- …

Content truncated to 500 words