How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710–721 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 710 How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions Goran Glavaš1, Robert Litschko1, Sebastian Ruder2,3∗, and Ivan Vuli´c4 1Data and Web Science Group, University of Mannheim, Germany 2Insight Research Centre, National University of Ireland, Galway, Ireland 3Aylien Ltd., Dublin, Ireland 4 PolyAI Ltd., London, United Kingdom {goran, litschko}@informatik.uni-mannheim.de, sebastian@ruder.io, iv250@cam.ac.uk Abstract Cross-lingual word embeddings (CLEs) facili- tate cross-lingual transfer of NLP models. De- spite their ubiquitous downstream usage, in- creasingly popular projection-based CLE mod- els are almost exclusively evaluated on bilin- gual lexicon induction (BLI). Even the BLI evaluations vary greatly, hindering our ability to correctly interpret performance and prop- erties of different CLE models. In this work, we take the first step towards a comprehensive evaluation of CLE models: we thoroughly eval- uate both supervised and unsupervised CLE models, for a large number of language pairs, on BLI and three downstream tasks, providing new insights concerning the ability of cutting- edge CLE models to support cross-lingual NLP. We empirically demonstrate that the per- formance of CLE models largely depends on the task at hand and that optimizing CLE mod- els for BLI may hurt downstream performance. We indicate the most robust supervised and unsupervised CLE models and emphasize the need to reassess simple baselines, which still display competitive performance across the board. We hope our work catalyzes further re- search on CLE evaluation and model analysis. 1 Introduction and Motivation Following the ubiquitous use of word embeddings in monolingual NLP tasks, research in word repre- sentation quickly broadened towards cross-lingual word embeddings (CLEs). CLE models learn vec- tors of words in two or more languages and rep- resent them in a shared cross-lingual word vector space where words with similar meanings obtain similar vectors, irrespective of their language. Ow- ing to this property, CLEs hold promise to support cross-lingual NLP by enabling multilingual model- ing of meaning and facilitating cross-lingual trans- fer for downstream NLP tasks and under-resourced ⋆Sebastian is now affiliated with DeepMind. languages. CLEs are used as (cross-lingual) knowl- edge sources in range of tasks, such as bilingual lexicon induction (Mikolov et al., 2013), docu- ment classification (Klementiev et al., 2012), infor- mation retrieval (Vuli´c and Moens, 2015), depen- dency parsing (Guo et al., 2015), sequence labeling (Zhang et al., 2016; Mayhew et al., 2017), and ma- chine translation (Artetxe et al., 2018c; Lample et al., 2018), among others. Earlier work typically induces CLEs by leverag- ing bilingual supervision from multilingual corpora aligned at the level of sentences (Zou et al., 2013; Hermann and Blunsom, 2014; Luong et al., 2015, inter alia) and documents (Søgaard et al., 2015; Vuli´c and Moens, 2016; Levy et al., 2017, inter alia). A recent trend are the so-called projection- based CLE models1, which post-hoc align pre- trained monolingual embeddings. Their popular- ity stems from competitive performance coupled …

Content truncated to 500 words