Estimating Mutual Information Between Dense Word Embeddings

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8361–8371 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8361 Estimating Mutual Information Between Dense Word Embeddings Vitalii Zhelezniak, Aleksandar Savkov & Nils Hammerla Babylon Health {firstname.lastname}@babylonhealth.com Abstract Word embedding-based similarity measures are currently among the top-performing meth- ods on unsupervised semantic textual similar- ity (STS) tasks. Recent work has increasingly adopted a statistical view on these embeddings, with some of the top approaches being essen- tially various correlations (which include the famous cosine similarity). Another excellent candidate for a similarity measure is mutual information (MI), which can capture arbitrary dependencies between the variables and has a simple and intuitive expression. Unfortunately, its use in the context of dense word embed- dings has so far been avoided due to difficul- ties with estimating MI for continuous data. In this work we go through a vast literature on estimating MI in such cases and single out the most promising methods, yielding a simple and elegant similarity measure for word em- beddings. We show that mutual information is a viable alternative to correlations, gives an excellent signal that correlates well with hu- man judgements of similarity and rivals exist- ing state-of-the-art unsupervised methods. 1 Introduction Neural text embeddings learned from unlabeled data are a key component of modern approaches to semantic textual similarity (STS). Despite the impressive performance of large pretrained models (Kiros et al., 2015; Conneau et al., 2017; Subra- manian et al., 2018; Cer et al., 2018; Peters et al., 2018; Radford, 2018; Devlin et al., 2018; Dai et al., 2019; Yang et al., 2019a) on a a plethora of hard NLP tasks, deep models do not currently offer a clear advantage over much simpler static word em- beddings (Bengio et al., 2003; Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017; Joulin et al., 2017) on standard unsupervised STS benchmarks (Hill et al., 2016; Arora et al., 2017; Wieting et al., 2016; Wieting and Gimpel, 2018; Zhelezniak et al., 2019b,a,c). Instead, the main sources of improvement here have come from train- ing on supervised paraphrastic corpora (Wieting et al., 2015, 2016; Wieting and Gimpel, 2018), de- signing better composition functions (Mitchell and Lapata, 2008; De Boom et al., 2016; Arora et al., 2017; Zhao and Mao, 2017; R¨uckl´e et al., 2018; Zhelezniak et al., 2019b,c; Yang et al., 2019b) and exploring novel similarity measures between word embeddings, in particular those inspired by opti- mal transport (Kusner et al., 2015; Huang et al., 2016), soft and fuzzy sets (Jimenez et al., 2010, 2015; Zhelezniak et al., 2019b), and statistics (Lev et al., 2015; Nikolentzos et al., 2017; Torki, 2018; Zhelezniak et al., 2019a,c). Recently, Zhelezniak et al. (2019a,c) advocated for a new statistical perspective on word embed- dings where each word embedding itself is viewed as a sample of (e.g. 300) observations from some scalar random variable. They conducted a statis- tical analysis of several popular pretrained word embeddings and their compositions …

Content truncated to 500 words