Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3336–3341, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3336 Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction Yova Kementchedjhieva University of Copenhagen yova@di.ku.dk Mareike Hartmann University of Copenhagen hartmann@di.ku.dk Anders Søgaard University of Copenhagen soegaard@di.ku.dk Abstract The task of bilingual dictionary induction (BDI) is commonly used for intrinsic evalua- tion of cross-lingual word embeddings. The largest dataset for BDI was generated auto- matically, so its quality is dubious. We study the composition and quality of the test sets for five diverse languages from this dataset, with concerning findings: (1) a quarter of the data consists of proper nouns, which can be hardly indicative of BDI performance, and (2) there are pervasive gaps in the gold-standard targets. These issues appear to affect the ranking be- tween cross-lingual embedding systems on in- dividual languages, and the overall degree to which the systems differ in performance. With proper nouns removed from the data, the mar- gin between the top two systems included in the study grows from 3.4% to 17.2%. Man- ual verification of the predictions, on the other hand, reveals that gaps in the gold standard tar- gets artificially inflate the margin between the two systems on English to Bulgarian BDI from 0.1% to 6.7%. We thus suggest that future re- search either avoids drawing conclusions from quantitative results on this BDI dataset, or ac- companies such evaluation with rigorous error analysis. 1 Introduction Bilingual dictionary induction (BDI) refers to re- trieving translations of individual words. The task has been widely used for intrinsic evalua- tion of cross-lingual embedding algorithms, which aim to map two languages into the same embed- ding space, for transfer learning purposes (Kle- mentiev et al., 2012). Recently, Glavas et al. (2019) reported limited evidence in support of this practice—they found that cross-lingual em- beddings optimized for a BDI evaluation metric were not necessarily better on downstream tasks. Here, we study BDI evaluation in itself, as has been done for other evaluation methods in the past (cf. Faruqui et al., 2016’s work on word similar- ity), with concerning findings about its reliability. A massive dataset of 110 bilingual dictionar- ies, known as the MUSE dataset, was introduced in early 2018 along with a strong baseline (Con- neau et al., 2018). Subsets of the MUSE dictio- naries have been used for model comparison in the evaluation of numerous cross-lingual embed- ding systems developed since (cf. Grave et al., 2018; Jawanpuria et al., 2018; Hoshen and Wolf, 2018a,b; Wada and Iwata, 2018; Joulin et al., 2018). Even though the field has been very ac- tive, progress has been incremental for most lan- guage pairs. Moreover, there have been very few attempts at a linguistically-informed error analy- sis of BDI performance as measured on MUSE (cf. Kementchedjhieva et al., 2018). This is problem- atic for two reasons: on one hand, most systems greatly …

Content truncated to 500 words