A Closer Look on Unsupervised Cross-lingual Word Embeddings Mapping

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5555–5562 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5555 A Closer Look on Unsupervised Cross-lingual Word Embeddings Mapping Kamil Pluciński1,2, Mateusz Lango1,2, Michał Zimniewicz1 1 Poznan Supercomputing and Networking Center, Poznań, Poland 2 Institute of Computer Science, Poznan University of Technology, Poznań, Poland kamil.plucinski97@gmail.com, mlango@cs.put.poznan.pl, mzimniewicz@man.poznan.pl Abstract In this work, we study the unsupervised cross-lingual word embeddings mapping method presented by Artetxe et al. (2018). First, we successfully reproduced the experiments performed in the original work, finding only minor differences. Furthermore, we verified the method’s robustness on different embedding representations and new language pairs, particularly these involving Slavic languages like Polish or Czech. We also performed an experimental analysis of the impact of the method’s parameters on the final result. Finally, we looked for an alternative way of initialization, which directly relies on the isometric assumption. Our work confirms the results presented earlier, at the same time pointing at interesting problems occurring while using the method with different types of embeddings or on less-common language pairs. Keywords: word embeddings, unsupervised word embedding mapping, cross-lingual mapping, bilingual lexicon construction 1. Introduction Adapting current NLP technologies to a larger set of lan- guages is an important challenge for both industry and academia. One of the common barriers in applying machine learning approaches to new languages is the lack of proper datasets. Fortunately, this issue can be sometimes circum- vented through transfer learning techniques. For instance, such techniques can enable a system trained on dataset for one language to perform well on the same task for another language (Ni et al., 2017). Such modern transfer tech- niques frequently leverage word embeddings as a medium for knowledge transfer. A word embedding is a representation of a word as a mul- tidimensional vector whose position in the space reflects semantic and/or syntactic similarities to other words in the same language. The majority of methods for the forma- tion of word embeddings require only a single, unannotated corpus, which fosters their usage on a wide range of lan- guages (Grave et al., 2018). Although word embeddings are available for many languages, their availability does not completely solve the problem of cross-lingual transfer. The embeddings for different languages are trained separately on distinct corpora, so they do not share the same vector space. This means that words representations with similar meaning in distinct languages can be very different. Not surprisingly, methods aligning word embeddings from dif- ferent languages have been proposed. One of the first such approaches was proposed by Mikolov et al. (2013b) and it used stochastic gradient descent to learn a translation matrix which mapped vectors from one space to another i.e. it mapped words from one language to another. The method was supervised and required a dataset of about five thousand pairs of translated words to learn the mapping. This still could be a problem for some languages. Later, it was noticed that learning a …

Content truncated to 500 words