Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 184–193 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 184 Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces Barun Patra⇤, Joel Ruben Antony Moniz⇤, Sarthak Garg⇤, Matthew R. Gormley, Graham Neubig Carnegie Mellon University {bpatra, jrmoniz, sarthakg, mgormley, gneubig}@cs.cmu.edu Abstract Recent work on bilingual lexicon induc- tion (BLI) has frequently depended either on aligned bilingual lexicons or on distribution matching, often with an assumption about the isometry of the two spaces. We propose a technique to quantitatively estimate this as- sumption of the isometry between two embed- ding spaces and empirically show that this as- sumption weakens as the languages in ques- tion become increasingly etymologically dis- tant. We then propose Bilingual Lexicon In- duction with Semi-Supervision (BLISS) — a semi-supervised approach that relaxes the iso- metric assumption while leveraging both lim- ited aligned bilingual lexicons and a larger set of unaligned word embeddings, as well as a novel hubness filtering technique. Our pro- posed method obtains state of the art results on 15 of 18 language pairs on the MUSE dataset, and does particularly well when the embed- ding spaces don’t appear to be isometric. In addition, we also show that adding supervision stabilizes the learning procedure, and is effec- tive even with minimal supervision.⇤ 1 Introduction Bilingual lexicon induction (BLI), the task of find- ing corresponding words in two languages from comparable corpora (Haghighi et al., 2008; Xing et al., 2015; Zhang et al., 2017a; Artetxe et al., 2017; Lample et al., 2018), finds use in numerous NLP tasks like POS tagging (Zhang et al., 2016), parsing (Xiao and Guo, 2014), document classi- fication (Klementiev et al., 2012), and machine translation (Irvine and Callison-Burch, 2013; Qi et al., 2018). Most work on BLI uses methods that learn a mapping between two word embedding spaces ⇤Equal Contribution ⇤Code to replicate the experiments presented in this work can be found at https://github.com/joelmoniz/ BLISS. (Ruder, 2017), which makes it possible to leverage pre-trained embeddings learned on large monolin- gual corpora. A commonly used method for BLI, which is also empirically effective, involves learn- ing an orthogonal mapping between the two em- bedding spaces (Mikolov et al. (2013a), Xing et al. (2015), Artetxe et al. (2016), Smith et al. (2017)). However, learning an orthogonal mapping inher- ently assumes that the embedding spaces for the two languages are isometric (subsequently re- ferred to as the orthogonality assumption). This is a particularly strong assumption that may not necessarily hold true, and consequently we can ex- pect methods relying on this assumption to pro- vide sub-optimal results. In this work, we exam- ine this assumption, identify where it breaks down, and propose a method to alleviate this problem. We first present a theoretically motivated ap- proach based on the Gromov-Hausdroff (GH) dis- tance to check the extent to which the orthogo- nality assumption holds (§2). We show that the constraint indeed does not hold, particularly for …

Content truncated to 500 words