Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4407–4418, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4407 Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? Ivan Vuli´c1, Goran Glavaš2, Roi Reichart3, Anna Korhonen1 1 Language Technology Lab, University of Cambridge 2 Data and Web Science Group, University of Mannheim 3 Faculty of Industrial Engineering and Management, Technion, IIT {iv250,alk23}@cam.ac.uk goran@informatik.uni-mannheim.de roiri@ie.technion.ac.il Abstract Recent efforts in cross-lingual word embed- ding (CLWE) learning have predominantly fo- cused on fully unsupervised approaches that project monolingual embeddings into a shared cross-lingual space without any cross-lingual signal. The lack of any supervision makes such approaches conceptually attractive. Yet, their only core difference from (weakly) super- vised projection-based CLWE methods is in the way they obtain a seed dictionary used to initialize an iterative self-learning procedure. The fully unsupervised methods have arguably become more robust, and their primary use case is CLWE induction for pairs of resource- poor and distant languages. In this paper, we question the ability of even the most robust un- supervised CLWE approaches to induce mean- ingful CLWEs in these more challenging set- tings. A series of bilingual lexicon induction (BLI) experiments with 15 diverse languages (210 language pairs) show that fully unsuper- vised CLWE methods still fail for a large num- ber of language pairs (e.g., they yield zero BLI performance for 87/210 pairs). Even when they succeed, they never surpass the perfor- mance of weakly supervised methods (seeded with 500-1,000 translation pairs) using the same self-learning procedure in any BLI setup, and the gaps are often substantial. These find- ings call for revisiting the main motivations be- hind fully unsupervised CLWE methods. 1 Introduction and Motivation The wide use and success of monolingual word embeddings in NLP tasks (Turian et al., 2010; Chen and Manning, 2014) has inspired further research focus on the induction of cross-lingual word embeddings (CLWEs). CLWE methods learn a shared cross-lingual word vector space where words with similar meanings obtain similar vectors regardless of their actual language. CLWEs benefit cross-lingual NLP, enabling multilingual modeling of meaning and supporting cross-lingual transfer for downstream tasks and resource-lean languages. CLWEs provide invaluable cross-lingual knowl- edge for, inter alia, bilingual lexicon induction (Gouws et al., 2015; Heyman et al., 2017), infor- mation retrieval (Vuli´c and Moens, 2015; Litschko et al., 2019), machine translation (Artetxe et al., 2018c; Lample et al., 2018b), document classifica- tion (Klementiev et al., 2012), cross-lingual plagia- rism detection (Glavaš et al., 2018), domain adapta- tion (Ziser and Reichart, 2018), cross-lingual POS tagging (Gouws and Søgaard, 2015; Zhang et al., 2016), and cross-lingual dependency parsing (Guo et al., 2015; Søgaard et al., 2015). The landscape of CLWE methods has recently been dominated by the so-called projection-based methods (Mikolov et al., 2013a; Ruder et al., 2019; Glavaš et al., 2019). They align two mono- lingual embedding spaces by learning a projec- tion/mapping based on a training dictionary of translation …

Content truncated to 500 words