Hubless Nearest Neighbor Search for Bilingual Lexicon Induction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4072–4080 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4072 Hubless Nearest Neighbor Search for Bilingual Lexicon Induction Jiaji Huang∗ Qiang Qiu† Kenneth Church∗ ∗ Baidu Research, Sunnyvale, CA, 94089 † Department of Electrical Engineering, Duke University, Durham, NC, 27707 ∗{huangjiaji, kennethchurch}@baidu.com †qiang.qiu@duke.edu Abstract Bilingual Lexicon Induction (BLI) is the task of translating words from corpora in two lan- guages. Recent advances in BLI work by aligning the two word embedding spaces. Fol- lowing that, a key step is to retrieve the near- est neighbor (NN) in the target space given the source word. However, a phenomenon called hubness often degrades the accuracy of NN. Hubness appears as some data points, called hubs, being extra-ordinarily close to many of the other data points. Reducing hubness is necessary for retrieval tasks. One successful example is Inverted SoFtmax (ISF), recently proposed to improve NN. This work pro- poses a new method, Hubless Nearest Neigh- bor (HNN), to mitigate hubness. HNN dif- fers from NN by imposing an additional equal preference assumption. Moreover, the HNN formulation explains why ISF works as well as it does. Empirical results demonstrate that HNN outperforms NN, ISF and other state-of- the-art. For reproducibility and follow-ups, we have published all code1. 1 Introduction This paper presents a new method for Bilingual Lexicon Induction (BLI), which we call Hub- less Nearest Neighbor (HNN). BLI is the task of creating a lexicon of translation equivalents such as, bank:banc or bank:banque automatically from non-parallel corpora. The proposed method not only improves upon but also unifies several recent works that retrieve translations by Near- est Neighbor (NN) (Mikolov et al., 2013) and more advanced techniques like Inverted SoFtmax (ISF) (Smith et al., 2017). There is a long history of BLI using non- parallel corpora. Methods often rely on some designed features, which reveal some shared 1https://github.com/baidu-research/HNN structures across languages. For example, co- occurrence matrices (Rapp, 1995), tf-idf of con- text words (Fung, 1998), and lexical similari- ties (Klementiev et al., 2012). A comprehensive study can be found in (Irvine and Callison-Burch, 2017). Recently, Mikolov et al. (2013) observe that iso- morphism exists across word embeddings of dif- ferent languages. This motivates them to learn a linear mapping to align the spaces, using a seed- ing dictionary of 5K pairs of translations. Af- ter that, more translations can be induced by NN search. Following the seminal work, significant advances have been made. For example, Faruqui and Dyer (2014) use Canonical Component Anal- ysis to align the two embedding spaces. Xing et al. (2015) show a substantial gain by normaliz- ing the embeddings and constraining the mapping to be orthogonal. A series of works by Artetxe et al. (2017, 2018a,b) show that decent accura- cies can be achieved even with a tiny or no seed- ing dictionary. The authors name their method as “self-learning”, which alternates between learning the mapping and inducing more translation pairs. The …

Content truncated to 500 words