Correlations between Word Vector Sets

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 77–87, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 77 Correlations between Word Vector Sets Vitalii Zhelezniak, April Shen, Daniel Busbridge, Aleksandar Savkov, Nils Hammerla {firstname.lastname} @babylonhealth.com Abstract Similarity measures based purely on word embeddings are comfortably competing with much more sophisticated deep learning and expert-engineered systems on unsupervised semantic textual similarity (STS) tasks. In contrast to commonly used geometric ap- proaches, we treat a single word embedding as e.g. 300 observations from a scalar ran- dom variable. Using this paradigm, we first il- lustrate that similarities derived from elemen- tary pooling operations and classic correla- tion coefficients yield excellent results on stan- dard STS benchmarks, outperforming many recently proposed methods while being much faster and trivial to implement. Next, we demonstrate how to avoid pooling operations altogether and compare sets of word embed- dings directly via correlation operators be- tween reproducing kernel Hilbert spaces. Just like cosine similarity is used to compare in- dividual word vectors, we introduce a novel application of the centered kernel alignment (CKA) as a natural generalisation of squared cosine similarity for sets of word vectors. Likewise, CKA is very easy to implement and enjoys very strong empirical results. 1 Introduction Distributed representations of text have had a mas- sive impact on the natural language processing (NLP), information retrieval (IR), and machine learning (ML) communities, thanks in part to their ability to capture rich notions of semantic sim- ilarity. While this work originally began with word embeddings (Bengio et al., 2003; Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017; Joulin et al., 2017), there is now an ever-increasing number of representations for longer units of text based on simple aggrega- tions of word vectors (Mitchell and Lapata, 2008; De Boom et al., 2016; Arora et al., 2017; Wieting et al., 2016; Wieting and Gimpel, 2018; Zhelez- niak et al., 2019b) as well as complex neural ar- chitectures (Le and Mikolov, 2014; Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Gan et al., 2017; Tang et al., 2017; Pagliardini et al., 2018; Zhelezniak et al., 2018; Subramanian et al., 2018; Cer et al., 2018; Devlin et al., 2018). By contrast, relatively little effort has been di- rected towards understanding the similarity mea- sures used to compare these textual embeddings, for which cosine similarity remains a convenient and widespread, yet somewhat arbitrary default, despite some emerging research into the alterna- tives (Camacho-Collados et al., 2015; De Boom et al., 2015; Santus et al., 2018; Zhelezniak et al., 2019b,a). Part of the appeal of cosine similarity perhaps lies in the simple geometric interpreta- tion behind it. However, as embeddings are ul- timately just arrays of numbers, we are free to take alternative viewpoints other than the geomet- ric ones, if they lead to illuminating insights or strong-performing methods. Following Zhelezniak et al. (2019a), we …

Content truncated to 500 words