Interventional Probing in High Dimensions: An NLI Case Study

Findings of the Association for Computational Linguistics: EACL 2023, pages 2489–2500 May 2-6, 2023 ©2023 Association for Computational Linguistics Interventional Probing in High Dimensions: An NLI Case Study Julia Rozanova1, Marco Valentino2, Lucas Cordeiro1, André Freitas1,2 University of Manchester, United Kingdom1 {firstname.lastname}@manchester.ac.uk1 Idiap Research Institute, Switzerland2 {firstname.lastname}@idiap.ch2 Abstract Probing strategies have been shown to detect the presence of various linguistic features in large language models; in particular, seman- tic features intermediate to the “natural logic" fragment of the Natural Language Inference task (NLI). In the case of natural logic, the rela- tion between the intermediate features and the entailment label is explicitly known: as such, this provides a ripe setting for interventional studies on the NLI models’ representations, al- lowing for stronger causal conjectures and a deeper critical analysis of interventional prob- ing methods. In this work, we carry out new and existing representation-level interventions to investigate the effect of these semantic fea- tures on NLI classification: we perform am- nesic probing (which removes features as di- rected by learned linear probes) and introduce the mnestic probing variation (which forgets all dimensions except the probe-selected ones). Furthermore, we delve into the limitations of these methods and outline some pitfalls have been obscuring the effectivity of interventional probing studies. 1 Introduction The probing paradigm has emerged as a useful in- terpretability methodology which has been shown to have reasonable information-theoretic underpin- nings (Pimentel et al., 2020; Voita and Titov, 2020; Zhu and Rudzicz, 2020), indicating whether a given feature is captured in the intermediate vector rep- resentations of neural models. It has been noted many times that this does not generally imply that the models are using these learnt features, and they may represent vestigial information from earlier training steps (Ravichander et al., 2021; Elazar et al., 2020). Only through interventional analyses can we start to make claims about which modelled fea- tures are used for a given downstream task: this is the aim of works such as Elazar et al. (2020); Model Layers Final Representation (CLS Token) NLI Model Classification Head Tokenized Text Input Intervention Method Iterative Nullspace Projection Train Probe for Auxiliary Concept Projection Matrix From Linear Probe Weights Repeat Until Probe Accuracy Reaches Baseline Projected Representation Auxiliary Concept Labels Intervened Representation Model Prediction NLI Evaluation Pre-Intervention Performance Post-Intervention Performance Figure 1: Workflow for interventional probing for NLP classification models: a basis for both the amnesic and mnestic intervention strategies. Giulianelli et al. (2018) and Geiger et al. (2021). We refer to the case where the interventions are guided by trained probes as interventional probing. It has been suggested in Elazar et al. (2020) (as the guidance for their amnesic probing methodol- ogy) that if features are strongly detected by probes, one may use debiasing methods such as iterative nullspace projection (INLP) (Ravfogel et al., 2020) to intervene on the corresponding vector representa- tions and effectively “remove" the features before re-insertion into the given classifier. Investigat- ing the effect of these intervention operations on the classifier performance could allow …

Content truncated to 500 words