Learning Unsupervised Multilingual Word Embeddings with Incremental Multilingual Hubs

Proceedings of NAACL-HLT 2019, pages 1890–1902 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1890 Learning Unsupervised Multilingual Word Embeddings with Incremental Multilingual Hubs Geert Heyman†, Bregt Verreet†, Ivan Vuli´c‡, Marie-Francine Moens† † LIIR, Department of Computer Science, KU Leuven ‡ Language Technology Lab, TAL, University of Cambridge {geert.heyman,bregt.verreet,sien.moens}@kuleuven.be iv250@cam.ac.uk Abstract Recent research has discovered that a shared bilingual word embedding space can be in- duced by projecting monolingual word embed- ding spaces from two languages using a self- learning paradigm without any bilingual super- vision. However, it has also been shown that for distant language pairs such fully unsuper- vised self-learning methods are unstable and often get stuck in poor local optima due to re- duced isomorphism between starting monolin- gual spaces. In this work, we propose a new ro- bust framework for learning unsupervised mul- tilingual word embeddings that mitigates the instability issues. We learn a shared multilin- gual embedding space for a variable number of languages by incrementally adding new lan- guages one by one to the current multilingual space. Through the gradual language addition our method can leverage the interdependencies between the new language and all other lan- guages in the current multilingual hub/space. We find that it is beneficial to project more distant languages later in the iterative pro- cess. Our fully unsupervised multilingual em- bedding spaces yield results that are on par with the state-of-the-art methods in the bilin- gual lexicon induction (BLI) task, and simul- taneously obtain state-of-the-art scores on two downstream tasks: multilingual document clas- sification and multilingual dependency pars- ing, outperforming even supervised baselines. This finding also accentuates the need to es- tablish evaluation protocols for cross-lingual word embeddings beyond the omnipresent in- trinsic BLI task in future work. 1 Introduction The ubiquitous use and success of word embed- dings in monolingual tasks inspired further re- search on inducing cross-lingual word embeddings for two or more languages in the same vector space. Embeddings of translations and words with simi- lar meaning are geometrically close in the shared cross-lingual vector space. This property makes them effective features for cross-lingual NLP tasks such as cross-lingual document classification (Kle- mentiev et al., 2012), cross-lingual information re- trieval (Vuli´c and Moens, 2015), bilingual lexicon induction (Mikolov et al., 2013b; Gouws et al., 2015; Heyman et al., 2017), and (unsupervised) machine translation (Artetxe et al., 2017b; Lample et al., 2018; Artetxe et al., 2018c). Most prior work has focused on methods for constructing bilingual word embeddings (BWEs), yielding word representations for exactly two lan- guages. For problems such as multilingual docu- ment classification, however, it is highly-desirable to represent words in a multilingual space. A favourable property is that it enables fitting a single classifier on the union of training datasets in many languages, which results in 1) knowledge transfer across languages that may lead to better classifica- tion performance, and 2) a setup that is easier to maintain as it is no longer required to train many different monolingual or …

Content truncated to 500 words