Adversarial Concept Erasure in Kernel Space

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6034 - 6055 December 7-11, 2022 ©2022 Association for Computational Linguistics Kernelized Concept Erasure Shauli Ravfogel1,2 Francisco Vargas3 Yoav Goldberg1,2 Ryan Cotterell4 1Bar-Ilan University 2Allen Institute for Artificial Intelligence 3University of Cambridge 4ETH Zürich {shauli.ravfogel, yoav.goldberg}@gmail.com fav25@cam.ac.uk ryan.cotterell@inf.ethz.ch Abstract The representation space of neural models for textual data emerges in an unsupervised man- ner during training. Understanding how those representations encode human-interpretable concepts is a fundamental problem. One promi- nent approach for the identification of concepts in neural representations is searching for a linear subspace whose erasure prevents the pre- diction of the concept from the representations. However, while many linear erasure algorithms are tractable and interpretable, neural networks do not necessarily represent concepts in a linear manner. To identify non-linearly encoded concepts, we propose a kernelization of a linear minimax game for concept erasure. We demonstrate that it is possible to prevent specific nonlinear adversaries from predicting the concept. However, the protection does not transfer to different nonlinear adversaries. Therefore, exhaustively erasing a non-linearly encoded concept remains an open problem. https://github.com/shauli-ravfogel/ adv-kernel-removal 1 Introduction Large neural networks in NLP produce real-valued representations that encode the bit of human language that they were trained on, e.g., words, sentences, or text grounded in images. For instance, GloVe (Pennington et al., 2014) produces real- valued representations of isolated words, BERT (Devlin et al., 2019) produces real-valued repre- sentations of sentences, and VilBERT produces real-valued representations of visually grounded language (Lu et al., 2019; Bugliarello et al., 2021). These real-valued representations naturally encode various properties of the objects they represent. For instance, a good representation of the first author’s laptop computer ought to encode its manufacturer, size, and color somewhere among its real values. We now describe the premise of our paper in more detail. We adopt the notion of a concept due to Gärdenfors (2000). For Gärdenfors, objects can be thought of as having a geometric representation. Different dimensions in the representation space that objects inhabit might correspond to their color, size, and shape. Gärdenfors then goes further and defines a concept as a convex region of the repre- sentation space.1 Building on Gärdenfors’ notion of a concept, this paper studies a task that we refer to as concept erasure. We now motivate the task more formally by extending the example. Imagine we have N different laptops, whose real-valued repre- sentations are denoted as x1, . . . , xN. Now, consider concept labels y1, . . . , yN that encode each laptop’s color and are taken from the set {GREY, SILVER, BLACK, WHITE}. In the concept erasure paradigm, we seek an erasure function r(·) such that r(x1), . . . , r(xN) are no longer predictive of the colors of the laptops, y1, . . . , yN, but retain all the other information encoded in the original representations, i.e., they remain predictive with respect to other laptop-related concepts. Our hope is that the geometry of the …

Content truncated to 500 words