Improving Bilingual Lexicon Induction for Low Frequency Words

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1310–1314, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1310 Improving Bilingual Lexicon Induction for Low Frequency Words Jiaji Huang Xingyu Cai Kenneth Church Baidu Research, 1195 Bordeaux Dr, Sunnyvale, CA, USA, 94089 {huangjiaji, xingyucai, kennethchurch}@baidu.com Abstract This paper designs a Monolingual Lexicon In- duction task and observes that two factors ac- company the degraded accuracy of bilingual lexicon induction for rare words. First, a di- minishing margin between similarities in low frequency regime, and secondly, exacerbated hubness at low frequency. Based on the ob- servation, we further propose two methods to address these two factors, respectively. The larger issue is hubness. Addressing that im- proves induction accuracy significantly, espe- cially for low-frequency words. 1 Introduction Bilingual Lexicon Induction (BLI) studies how to generate word-level translations from non-parallel corpora in two languages. Recently, Irvine and Callison-Burch (2017) observe that rarer words are harder to translate than frequent ones. But their BLI method is based on various “hand-crafted” features. We show that the same phenomenon occurs as well in BLI methods that are based on word embeddings. This type of methods have be- come especially popular in recent years (Mikolov et al., 2013; Faruqui and Dyer, 2014; Artetxe et al., 2016, 2018) and achieved state-of-art accuracies (Conneau et al., 2018). We briefly review BLI methods that are based on word embeddings. Without loss of general- ity, in this paper, we focus on “supervised” BLI, which assumes that a seeding dictionary is avail- able. Unsupervised BLI (Artetxe et al., 2018; Con- neau et al., 2018) often alternates between induc- ing a seeding dictionary and using that to refine generated translations. Therefore, to some extent, “supervised” BLI is a key step in its “unsuper- vised” counterpart, and a more basic prototype to study. Let the source space be X ≜ [x1, . . . , xm], where xi ∈ Rd is the embedding vector for the i-th source word. Similarly, let the target space be Y ≜ [y1, . . . , yn] where yj is the embed- ding vector for the j-th target word. Here m and n are the vocabulary sizes for the two spaces. The seeding dictionary is made up of subsets of X and Y, denoted as Xs = [xs 1, . . . , xs S] and Ys = [ys 1, . . . , ys S] respectively, where xs k and ys are the word embeddings of a pair of translations.k S is the size of seeding dictionary. The typical supervised BLI works by first learn- ing a transformation W that minimizes the dis- crepancy between Xs and Ys, W = arg min W∈W ∥WXs − Ys∥2 F , (1) where ∥ · ∥F is Frobenius norm, and W is a con- straint set of W. The easiest choice of W may be Rd×d, seen in (Mikolov et al., 2013). On the other hand, Xing et al. (2015) has observed sub- stantial gain by …

Content truncated to 500 words