IsoScore: Measuring the Uniformity of Embedding Space Utilization

Findings of the Association for Computational Linguistics: ACL 2022, pages 3325 - 3339 May 22-27, 2022 c⃝2022 Association for Computational Linguistics IsoScore: Measuring the Uniformity of Embedding Space Utilization William Rudman†, Nate Gillman‡, Taylor Rayne∗, Carsten Eickhoff† Department of Computer Science, Brown University† Department of Mathematics, Brown University‡ Quest University∗ {william_rudman, ngillman, carsten}@brown.edu taylor.rayne@questu.ca Abstract The recent success of distributed word repre- sentations has led to an increased interest in analyzing the properties of their spatial distri- bution. Several studies have suggested that contextualized word embedding models do not isotropically project tokens into vector space. However, current methods designed to mea- sure isotropy, such as average random cosine similarity and the partition score, have not been thoroughly analyzed and are not appro- priate for measuring isotropy. We propose IsoScore: a novel tool that quantifies the de- gree to which a point cloud uniformly utilizes the ambient vector space. Using rigorously de- signed tests, we demonstrate that IsoScore is the only tool available in the literature that ac- curately measures how uniformly distributed variance is across dimensions in vector space. Additionally, we use IsoScore to challenge a number of recent conclusions in the NLP liter- ature that have been derived using brittle met- rics of isotropy. We caution future studies from using existing tools to measure isotropy in contextualized embedding space as result- ing conclusions will be misleading or alto- gether inaccurate. 1 Introduction & Background The first step in any natural language processing pipeline is to represent text in a vector space. Un- derstanding how contextualized word embedding models project tokens into vector space is crucial for advancing the field of natural language process- ing. Several recent studies analyzing the spatial dis- tribution of contextualized word embeddings claim that the point clouds induced by models such as BERT or GPT-2 do not uniformly utilize all dimen- sions of the vector space they occupy (Ethayarajh, 2019; Mickus et al., 2019; Cai et al., 2021; Coenen et al., 2019b; Gao et al., 2019). Figure 1 illustrates a two-dimensional disk that uniformly utilizes the x and y axes in two- dimensional space, but does not uniformly utilize all dimensions when embedded into three dimen- sions. Figure 1: From left to right, a line, disk, and ball em- bedded in 3D space. A distribution is isotropic when variance is uni- formly distributed across all dimensions. Namely, a distribution is fully isotropic when the covariance matrix is proportional to the identity matrix. Sev- eral authors suggest that isotropy correlates with improved performance of embedding models (Bi´s et al., 2021; Wang et al., 2019; Coenen et al., 2019a; Gong et al., 2018; Hasan and Curry, 2017; He- witt and Manning, 2019; Liang et al., 2021; Zhou et al., 2019, 2021). However, current methods of measuring the spatial utilization of contextualized embedding models do not truly measure isotropy. The most commonly used methods for measuring spatial distribution in embedding spaces include av- erage random cosine similarity, the partition score, variance explained and intrinsic dimensionality es- timation. In …

Content truncated to 500 words