Just Rank: Rethinking Evaluation with Word and Sentence Similarities

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6060 - 6077 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Just Rank: Rethinking Evaluation with Word and Sentence Similarities Bin Wang†, C.-C. Jay Kuo§, Haizhou Li‡,†, ♯ † National University of Singapore, Singapore § University of Southern California, USA ‡ The Chinese University of Hong Kong, Shenzhen, China ♯ Kriston AI, China bwang28c@gmail.com Abstract Word and sentence embeddings are useful fea- ture representations in natural language pro- cessing. However, intrinsic evaluation for em- beddings lags far behind, and there has been no significant update since the past decade. Word and sentence similarity tasks have be- come the de facto evaluation method. It leads models to overfit to such evaluations, nega- tively impacting embedding models’ develop- ment. This paper first points out the prob- lems using semantic similarity as the gold stan- dard for word and sentence embedding evalua- tions. Further, we propose a new intrinsic eval- uation method called EvalRank, which shows a much stronger correlation with downstream tasks. Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments. Finally, the practical evaluation toolkit is released for future bench- marking purposes.1 1 Introduction Distributed representation of words (Bengio et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) and sentences (Kiros et al., 2015; Conneau et al., 2017; Reimers and Gurevych, 2019; Gao et al., 2021) have shown to be extremely useful in transfer learning to many NLP tasks. Therefore, it plays an essential role in how we evaluate the quality of embedding models. Among many evaluation methods, the word and sentence similarity task gradually becomes the de facto intrinsic evaluation method. Figure 1 shows examples from word and sen- tence similarity datasets. In general, the datasets consist of pairs of words (w1, w2) (or sentences) and human-annotated similarity scores Sh. To eval- uate an embedding model φ(·), we first extract em- beddings for (w1,w2): (e1, e2) = (φ(w1), φ(w2)). 1Available at https://github.com/BinWang28/ EvalRank-Embedding-Evaluation. p1: Telephone p1: Communication p2: Butter p2: Bread p3: Cucumber p3: Professor 7.50 6.19 0.31 p4: A woman fires a shotgun p4: A woman is shooting a gun 6.40 p5: A sheep standing on a table at auction p5: Woolly sheep at auction house 8.40 Word Pairs Sentence Pairs Sim Scores Figure 1: Word and sentence pairs with human- annotated similarity scores from WS-353 and STS-B datasets (scaled to range 0 (lowest) to 10 (highest)). Then, a similarity measure is applied to compute an predicted score Sp = sim(e1, e2), where cosine similarity is adopted as sim unquestionably in the majority of cases. Finally, the correlation between Sh and Sp is computed, and a higher correlation suggests good alignment with human annotations and a better embedding model. Many studies, especially those targeting on infor- mation retrieval via semantic search and clustering (Reimers and Gurevych, 2019; Su et al., 2021), have used the similarity task as the only or …

Content truncated to 500 words