Non-Linearity in Mapping Based Cross-Lingual Word Embeddings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3583–3589 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3583 Non-Linearity in mapping based Cross-Lingual Word Embeddings Jiawei Zhao, Andrew Gilman Massey University Auckland, New Zealand {j.zhao1, a.gilman}@massey.ac.nz Abstract Recent works on cross-lingual word embeddings have been mainly focused on linear-mapping-based approaches, where pre-trained word embeddings are mapped into a shared vector space using a linear transformation. However, there is a limitation in such approaches–they follow a key assumption: words with similar meanings share similar geometric arrangements between their monolingual word embeddings, which suggest that there is a linear relationship between languages. However, such assumption may not hold for all language pairs across all semantic concepts. We investigate whether non-linear mappings can better describe the relationship between different languages by utilising kernel Canonical Correlation Analysis (KCCA). Experimental results on five language pairs show an improvement over current state-of-art results in both supervised and self-learning scenarios, confirming that non-linear mapping is a better way to describe the relationship between languages. Keywords: Cross-lingual Word Embedding, KCCA, Non-linearity 1. Introduction Cross-lingual representations have gained much interest re- cently. It has been shown that cross-lingual word embed- ding models succeed in many inherently cross-lingual Nat- ural Language Processing (NLP) tasks such as machine translation and cross-lingual entity linking (Artetxe et al., 2018c)(Lample et al., 2018)(Tsai and Roth, 2016). Cross- lingual word embedding models also allow to reason word semantics in multi-context environments and helps to trans- fer semantic knowledge from rich- to low-resource lan- guages. Linear-mapping-based cross-lingual word embeddings rely on a basic assumption, first stated by Mikolov et al. (2013), that words with similar meanings have a similar geomet- ric arrangement in the embedding vector space. However, this assumption may be heavily violated for languages with different cultural backgrounds. Therefore, we hypothesis that non-linearity can better express the relationships be- tween languages. Based on such hypothesis, we perform a non-linear alignment between word embeddings of differ- ent languages. In this paper, we introduce a KCCA-based mapping ap- proach, which can find non-linear relationships between languages. Our experiments show that the addition of non- linearity can improve cross-lingual word embeddings for a number of language pairs. Our code and the new English- Chinese dataset are released under open source license1. We review prior work in Section 2. In Section 3, we intro- duce our proposed approach. Section 4 presents our exper- iments and analysis of the experimental results is given in Section 5. Section 6 concludes our work. 2. Related Work The mapping-based approach was first proposed by Mikolov et al. (2013), who first train monolingual word embeddings for two languages independently, and then map 1https://gitlab.com/zjw1990/kclwe those embeddings into a shared vector space based by min- imising the Euclidean distance between embeddings repre- senting the same word in the two languages. Xing et al. (2015) also followed this approach, but argued that word embeddings should be normalised to unit length. Artetxe et al. (2016) further showed …

Content truncated to 500 words