All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4527–4546 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4527 All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality William Timkey and Marten van Schijndel Department of Linguistics Cornell University {wpt25|mv443}@cornell.edu Abstract Similarity measures are a vital tool for under- standing how language models represent and process language. Standard representational similarity measures such as cosine similarity and Euclidean distance have been successfully used in static word embedding models to un- derstand how words cluster in semantic space. Recently, these measures have been applied to embeddings from contextualized models such as BERT and GPT-2. In this work, we call into question the informativity of such mea- sures for contextualized language models. We find that a small number of rogue dimensions, often just 1-3, dominate these measures. More- over, we find a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model. We show that simple postpro- cessing techniques such as standardization are able to correct for rogue dimensions and reveal underlying representational quality. We argue that accounting for rogue dimensions is essen- tial for any similarity-based analysis of contex- tual language models. 1 Introduction By mapping words into continuous vector spaces, we can reason about human language in geomet- ric terms. For example, the cosine similarity of pairs of word embeddings in Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) shows a robust correlation with human similarity judgments, and embeddings cluster into natural se- mantic classes in Euclidean space (Baroni et al., 2014; Wang et al., 2019). In recent years, static embeddings have given way to their contextual counterparts, with language models based on the transformer architecture (Vaswani et al., 2017) such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2020), XLNet (Yang et al., 2019) and GPT-2 (Rad- ford et al., 2019) achieving state of the art results on many language understanding tasks. Despite their success, relatively little is known about how these models represent and process language. Recent work has employed measures such as cosine sim- ilarity and Euclidean distance to contextual repre- sentations with unclear and counterintuitive results. For example, similarity/distance measures in BERT are extremely sensitive to word position, leading to inconsistent results on evaluation benchmarks (Mickus et al., 2020; May et al., 2019). Addition- ally, representational quality appears to degrade severely in later layers of each network, with the final layers of BERT, RoBERTa, GPT-2 and XLNet showing little to no correlation with the semantic similarity/relatedness judgments of humans (Bom- masani et al., 2020). Recent work which probes the representational geometry of contextualized embedding spaces us- ing cosine similarity has found that contextual em- beddings have several counterintuitive properties (Ethayarajh, 2019). For example: 1) Word repre- sentations are highly anisotropic: randomly sam- pled words tend to be highly similar to one an- other when measured by cosine similarity. In the final layer of …

Content truncated to 500 words