The Low-Dimensional Linear Geometry of Contextualized Word Representations

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 82–93 November 10–11, 2021. ©2021 Association for Computational Linguistics 82 The Low-Dimensional Linear Geometry of Contextualized Word Representations Evan Hernandez MIT CSAIL dez@mit.edu Jacob Andreas MIT CSAIL jda@mit.edu Abstract Black-box probing models can reliably extract linguistic features like tense, number, and syn- tactic role from pretrained word representa- tions. However, the manner in which these fea- tures are encoded in representations remains poorly understood. We present a systematic study of the linear geometry of contextualized word representations in ELMO and BERT. We show that a variety of linguistic features (in- cluding structured dependency relationships) are encoded in low-dimensional subspaces. We then refine this geometric picture, showing that there are hierarchical relations between the subspaces encoding general linguistic cat- egories and more specific ones, and that low- dimensional feature encodings are distributed rather than aligned to individual neurons. Fi- nally, we demonstrate that these linear sub- spaces are causally related to model behavior, and can be used to perform fine-grained manip- ulation of BERT’s output distribution. 1 Introduction Contextual word representations (Peters et al., 2018) encode general linguistic features (e.g. se- mantic class; Belinkov and Glass, 2019) and sentence-specific relations (e.g. syntactic role; Ten- ney et al., 2019a). Used as input features, pre- trained representations enable efficient training of models for a variety of NLP tasks (Peters et al., 2019). An enormous body of recent work in NLP has attempted to enumerate what features are en- coded by pretrained word representations (e.g. Shi et al., 2016; Tenney et al., 2019b; Belinkov and Glass, 2019), and more recent approaches have studied how accessible this information is, char- acterizing the tradeoff between generic notions of model complexity and accuracy needed to recover word features from learned representations (Voita and Titov, 2020; Pimentel et al., 2020a). But the manner in which these features are encoded in rep- resentations remains poorly understood. Are subspaces encoding linguistic features… hierarchical? or unstructured? axis-aligned? or distributed? gerunds nouns proper nouns everything else f(x1) f(x0, x1) x0 x1 x0 x1 direct objects everything else He protested his firing . (a) (b) (c) MODEL ? low-dimensional? or full-rank? Low-Dim Linear Map everything else The cat [MASK] on the mat Nullspace Proj. verbs everything else dog sat causal? or correlative? (d) Figure 1: By training rank-constrained probing mod- els on linguistic analysis tasks, we find that linguis- tically meaningful categories are (a) encoded in low- dimensional representation spaces; (b) organized hier- archically, but (c) not aligned with individual neurons. An additional set of experiments (d) uses these findings to identify linear operators that predictably affect out- puts of a masked language model. Best viewed in color. In this paper, we investigate the shape and structure of learned representation spaces (Fig- ure 1). We build on previous studies that have identified specific linear subspaces responsible for gender bias in word embeddings (Bolukbasi et al., 2016) and word sense disambiguation (Co- enen et al., 2019). We show that these linear sub- …

Content truncated to 500 words