Are Girls Neko or Shōjo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3180–3189 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3180 Are Girls Neko or Sh¯ojo? Cross-Lingual Alignment of Non-Isomorphic Embeddings with Iterative Normalization Mozhi Zhang1 Keyulu Xu2 Ken-ichi Kawarabayashi3 Stefanie Jegelka2 Jordan Boyd-Graber1 1University of Maryland, College Park, Maryland, USA 2Massachusetts Institute of Technology, Cambridge, Massachusetts, USA 3National Institue of Informatics, Tokyo, Japan {mozhi,jbg}@umiacs.umd.edu {keyulu,stefje}@mit.edu k_keniti@nii.ac.jp Abstract Cross-lingual word embeddings (CLWE) un- derlie many multilingual natural language processing systems, often through orthogo- nal transformations of pre-trained monolin- gual embeddings. However, orthogonal map- ping only works on language pairs whose em- beddings are naturally isomorphic. For non- isomorphic pairs, our method (Iterative Nor- malization) transforms monolingual embed- dings to make orthogonal alignment easier by simultaneously enforcing that (1) individual word vectors are unit length, and (2) each lan- guage’s average vector is zero. Iterative Nor- malization consistently improves word trans- lation accuracy of three CLWE methods, with the largest improvement observed on English- Japanese (from 2% to 44% test accuracy). 1 Orthogonal Cross-Lingual Mappings Cross-lingual word embedding (CLWE) models map words from multiple languages to a shared vec- tor space, where words with similar meanings are close, regardless of language. CLWE is widely used in multilingual natural language processing (Kle- mentiev et al., 2012; Guo et al., 2015; Zhang et al., 2016). Recent CLWE methods (Ruder et al., 2017; Glavas et al., 2019) independently train two mono- lingual embeddings on large monolingual corpora and then align them with a linear transformation. Previous work argues that these transformations should be orthogonal (Xing et al., 2015; Smith et al., 2017; Artetxe et al., 2016): for any two words, the dot product of their representations is the same as the dot product with the transformation. This preserves similarities and substructure of the original monolingual word embedding but enriches the embeddings with multilingual connections be- tween languages. Thus, many state-of-the-art mapping-based CLWE methods impose an orthogonal con- straint (Artetxe et al., 2017; Conneau et al., 2018; Alvarez-Melis and Jaakkola, 2018; Artetxe et al., 2018; Ruder et al., 2018; Alvarez-Melis et al., 2019). The success of orthogonal methods relies on the assumption that embedding spaces are isomor- phic; i.e., they have the same inner-product struc- tures across languages, but this does not hold for all languages (Søgaard et al., 2018; Fujinuma et al., 2019). For example, English and Japanese fastText vectors (Bojanowski et al., 2017) have different substructures around “girl” (Figure 1 left). As a re- sult, orthogonal mapping fails on some languages— when Hoshen and Wolf (2018) align fastText em- beddings with orthogonal mappings, they report 81% English–Spanish word translation accuracy but only 2% for the more distant English–Japanese. While recent work challenges the orthogonal as- sumption (Doval et al., 2018; Joulin et al., 2018; Jawanpuria et al., 2019), we focus on whether sim- ple preprocessing techniques can improve the suit- ability of orthogonal models. Our iterative method normalizes monolingual embeddings to make their structures more similar …

Content truncated to 500 words