The Limitations of Cross-language Word Embeddings Evaluation

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 94–100 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics The Limitations of Cross-language Word Embeddings Evaluation Amir Bakarov†* Roman Suvorov* Ilya Sochenkov‡* †National Research University Higher School of Economics, *Federal Research Center ‘Computer Science and Control’ of the Russian Academy of Sciences, ‡Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia amirbakarov@gmail.com, rsuvorov@isa.ru, ivsochenkov@gmail.com Abstract The aim of this work is to explore the possi- ble limitations of existing methods of cross- language word embeddings evaluation, ad- dressing the lack of correlation between in- trinsic and extrinsic cross-language evaluation methods. To prove this hypothesis, we con- struct English-Russian datasets for extrinsic and intrinsic evaluation tasks and compare per- formances of 5 different cross-language mod- els on them. The results say that the scores even on different intrinsic benchmarks do not correlate to each other. We can conclude that the use of human references as ground truth for cross-language word embeddings is not proper unless one does not understand how do native speakers process semantics in their cognition. 1 Introduction Real-valued word representations called word em- beddings are an ubiquitous and effective technique of semantic modeling. So it is not surprising that cross-language extensions of such models (cross- language word embeddings) rapidly gained popu- larity in the NLP community (Vuli´c and Moens, 2013), proving their effectiveness in certain cross- language NLP tasks (Upadhyay et al., 2016). However, the problem of proper evaluation of any type of word embeddings still remains open. In recent years there was a critique to main- stream methods of intrinsic evaluation: some re- searchers addressed subjectivity of human assess- ments, obscurity of instructions for certain tasks and terminology confusions (Faruqui et al., 2016; Batchkarov et al., 2016). Despite all these limi- tations, some of the criticized methods (like the word similarity task) has been started to be ac- tively applied yet for cross-language word embed- dings evaluation (Camacho-Collados et al., 2017, 2015). We argue that if certain tasks are considered as not proper enough for mono-lingual evalua- tion, then it should be even more inappropriate to use them for cross-language evaluation since new problems would appear due to the new fea- tures of cross-linguality wherein the old limita- tions still remain. Moreover, it is still unknown for the field of cross-language word embeddings, are we able to make relevant predictions on per- formance of the model on one method, using an- other. We do not know whether can we use the relative ordering of different embeddings obtained by evaluation on an intrinsic task to decide which model will be better on a certain extrinsic task. So, the aim of this work is to highlight the lim- itations of cross-language intrinsic benchmarks, studying the connection of outcomes from differ- ent cross-language word embeddings evaluation schemes (intrinsic evaluation and extrinsic evalu- ation), and explain this connection by addressing certain issues of intrinsic benchmarks that hamper us to have a correlation between two evaluation schemes. …

Content truncated to 500 words