Correlation Coefficients and Semantic Textual Similarity

Proceedings of NAACL-HLT 2019, pages 951–962 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 951 Correlation Coefficients and Semantic Textual Similarity Vitalii Zhelezniak, Aleksandar Savkov, April Shen & Nils Y. Hammerla Babylon Health {firstname.lastname} @babylonhealth.com Abstract A large body of research into semantic tex- tual similarity has focused on constructing state-of-the-art embeddings using sophisti- cated modelling, careful choice of learning signals and many clever tricks. By contrast, little attention has been devoted to similar- ity measures between these embeddings, with cosine similarity being used unquestionably in the majority of cases. In this work, we illustrate that for all common word vectors, cosine similarity is essentially equivalent to the Pearson correlation coefficient, which pro- vides some justification for its use. We thor- oughly characterise cases where Pearson cor- relation (and thus cosine similarity) is unfit as similarity measure. Importantly, we show that Pearson correlation is appropriate for some word vectors but not others. When it is not appropriate, we illustrate how common non- parametric rank correlation coefficients can be used instead to significantly improve perfor- mance. We support our analysis with a se- ries of evaluations on word-level and sentence- level semantic textual similarity benchmarks. On the latter, we show that even the simplest averaged word vectors compared by rank cor- relation easily rival the strongest deep repre- sentations compared by cosine similarity. 1 Introduction Textual embeddings are immensely popular be- cause they help us reason about the abstract and fuzzy notion of semantic similarity in purely geo- metric terms. Distributed representations of words in particular (Bengio et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017; Joulin et al., 2017) have had a massive im- pact on machine learning (ML), natural language processing (NLP), and information retrieval (IR). Recently, much effort has also been directed to- wards learning representations for larger pieces of text, with methods ranging from clever composi- tions of word embeddings (Mitchell and Lapata, 2008; De Boom et al., 2016; Arora et al., 2017; Wieting et al., 2016; Wieting and Gimpel, 2018; Zhelezniak et al., 2019) to sophisticated neural ar- chitectures (Le and Mikolov, 2014; Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Gan et al., 2017; Tang et al., 2017; Zhelezniak et al., 2018; Subramanian et al., 2018; Pagliardini et al., 2018; Cer et al., 2018). Comparatively, there is little research into sim- ilarity measures for textual embeddings. Despite some investigations into alternatives (Camacho- Collados et al., 2015; De Boom et al., 2015; Santus et al., 2018; Zhelezniak et al., 2019), cosine simi- larity has persistently remained the default and un- questioned choice across the field. This is partly because cosine similarity is very convenient and easy to understand. Sometimes, however, we have to resist what is convenient and instead use what is appropriate. The core idea behind our work is to treat each word or sentence embedding as a sample of (e.g. 300) observations from some scalar ran- dom variable. Hence, …

Content truncated to 500 words