Aligning Vector-spaces with Noisy Supervised Lexicon

Proceedings of NAACL-HLT 2019, pages 460–465 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 460 Aligning Vector-spaces with Noisy Supervised Lexicons Noa Yehezkel Lubin†1 Jacob Goldberger‡2 Yoav Goldberg†∗3 †Computer Science Department, Bar Ilan University, Ramat Gan, Israel ‡Electrical Engineering Department, Bar Ilan University, Ramat Gan, Israel ∗Allen Institute for Artificial Intelligence 1noa.kel@gmail.com 2jacob.goldberger@biu.ac.il 3yoav.goldberg@gmail.com Abstract The problem of learning to translate between two vector spaces given a set of aligned points arises in several application areas of NLP. Cur- rent solutions assume that the lexicon which defines the alignment pairs is noise-free. We consider the case where the set of aligned points is allowed to contain an amount of noise, in the form of incorrect lexicon pairs and show that this arises in practice by ana- lyzing the edited dictionaries after the cleaning process. We demonstrate that such noise sub- stantially degrades the accuracy of the learned translation when using current methods. We propose a model that accounts for noisy pairs. This is achieved by introducing a generative model with a compatible iterative EM algo- rithm. The algorithm jointly learns the noise level in the lexicon, finds the set of noisy pairs, and learns the mapping between the spaces. We demonstrate the effectiveness of our proposed algorithm on two alignment problems: bilingual word embedding transla- tion, and mapping between diachronic embed- ding spaces for recovering the semantic shifts of words across time periods. 1 Introduction We consider the problem of mapping between points in different vector spaces. This problem has prominent applications in natural language pro- cessing (NLP). Some examples are creating bilin- gual word lexicons (Mikolov et al., 2013), ma- chine translation (Artetxe et al., 2016, 2017a,b, 2018a,b; Conneau et al., 2017), hypernym gen- eration (Yamane et al., 2016), diachronic embed- dings alignment (Hamilton et al., 2016) and do- main adaptation (Barnes et al., 2018). In all these examples one is given word embeddings in two different vector spaces, and needs to learn a map- ping from one to the other. The problem is traditionally posed as a super- vised learning problem, in which we are given two sets of vectors (e.g.: word-vectors in Italian and in English) and a lexicon mapping the points between the two sets (known word-translation pairs). Our goal is to learn a mapping that will correctly map the vectors in one space (e.g.: English word em- beddings) to their known corresponding vectors in the other (e.g.: Italian word embeddings). The mapping will then be used to translate vectors for which the correspondence is unknown. This setup was popularized by Mikolov et al. (2013). The supervised setup assumes a perfect lexicon. Here, we consider what happens in the presence of training noise, where some of the lexicon’s entries are incorrect in the sense that they don’t reflect an optimal correspondence between the word vectors. 2 Background 2.1 The Supervised Translation Problem We are given two datasets, X = x1, ..., xm and Y = y1, ..., yn, coming from d-dimensional …

Content truncated to 500 words