Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals Yanai Elazar1,2 Shauli Ravfogel1,2 Alon Jacovi1 Yoav Goldberg1,2 1Computer Science Department, Bar Ilan University 2Allen Institute for Artificial Intelligence {yanaiela,shauli.ravfogel,alonjacovi,yoav.goldberg}@gmail.com Abstract A growing body of work makes use of probing in order to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer beha- vioral conclusions from probing results, and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be as- sessed by measuring the influence of a causal intervention that removes it from the represen- tation. Equipped with this new analysis tool, we can ask questions that were not possible before, for example, is part-of-speech informa- tion important for word prediction? We per- form a series of analyses on BERT to answer these types of questions. Our findings demon- strate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.1 1 Introduction What drives a model to perform a specific pre- diction? What information is being used for prediction, and what would have happen if that information went missing? Because neural repre- sentation is opaque and hard to interpret, answer- ing these questions is challenging. The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b) spiked interest 1The code is available at: https://github.com /yanaiela/amnesic probing. in understanding how these models work and what is being encoded in them. One prominent meth- odology that attempts to shed light on those ques- tions is probing (Conneau et al., 2018) (also known as auxilliary prediction [Adi et al., 2016] anddiagnosticclassification [Hupkes et al., 2018]). Under this methodology, one trains a simple model —a probe—to predict some desired information from the latent representations of the pre-trained model. High prediction performance is interpreted as evidence for the information being encoded in the representation. A key drawback of such an approach is that while it may indicate that the information can be extracted from the represen- tation, it provides no evidence for or against the actual use of this information by the model. Indeed, Hewitt and Liang (2019) have shown that under certain conditions, above-random probing accuracy can be achieved even when the infor- mation that one probes for is linguistically mean- ingless noise, which is unlikely to have any use by the actual model. More recently, Ravichander et al. (2020) showed that models encode linguis- tic properties, even when not required at all for solving the task, questioning the usefulness and common interpretation of probing. These results call for higher scrutiny of causal claims based on probing results. In …

Content truncated to 500 words