Hierarchical Mapping for Crosslingual Word Embedding Alignment

Hierarchical Mapping for Crosslingual Word Embedding Alignment Ion Madrazo Azpiazu and Maria Soledad Pera Department of Computer Science Boise State University {ionmadrazo,solepera}@boisestate.edu Abstract The alignment of word embedding spaces in different languages into a common crosslin- gual space has recently been in vogue. Strate- gies that do so compute pairwise alignments and then map multiple languages to a single pivot language (most often English). These strategies, however, are biased towards the choice of the pivot language, given that language proximity and the linguistic char- acteristics of the target language can strongly impact the resultant crosslingual space in detri- ment of topologically distant languages. We present a strategy that eliminates the need for a pivot language by learning the mappings across languages in a hierarchical way. Experi- ments demonstrate that our strategy signifi- cantly improves vocabulary induction scores in all existing benchmarks, as well as in a new non-English–centered benchmark we built, which we make publicly available. 1 Introduction Word embeddings have changed how we build text processing applications, given their capabilities for representing the meaning of words (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017). Traditional embedding-generation strategies create different embeddings for the same word depending on the language. Even if the embeddings themselves are different across lan- guages, their distributions tend to be consistent— the relative distances across word embeddings are preserved regardless of the language (Mikolov et al., 2013b). This behavior has been exploited for crosslingual embedding generation by aligning any two monolingual embeddings spaces into one (Dinu et al., 2014; Xing et al., 2015; Artetxe et al., 2016). Alignment techniques have been successful in generating bilingual embedding spaces that can later be merged into a crosslingual space using a pivoting language, English being the most common choice. Unfortunately, mapping one language into another suffers from a neutrality problem, as the resultant bilingual space is impacted by language-specific phenomena and corpus-specific biases of the target language (Doval et al., 2018). To address this issue, Doval et al. (2018) propose mapping any two languages into a different middle space. This mapping, however, precludes the use of a pivot language for merging multiple bilingual spaces into a crosslingual one, limiting the solution to a bilingual scenario. Additionally, the pivoting strategy suffers from a generalized bias problem, as languages that are the most similar to the pivot obtain a better alignment and are therefore better representedin the crosslingual space. This is because language proximity is a key factor when learning alignments. This is evidenced by the results in Artetxe et al. (2017), which indicate that when using English (Indo-European) as a pivot, the vocabulary induction results for Finnish (Uralic) are about 10 points below the rest of the Indo-European languages under study. If we want to incorporate all languages into the same crosslingual space regardless of their characteristics, we need to go beyond the train- bilingual/merge-by-pivoting (TB/MP) model, and instead seek solutions that can directly generate crosslingual spaces without requiring a bilingual step. This motivates the design …

Content truncated to 500 words