Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1780–1790 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1780 Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models Christopher Grimsley1*, Elijah Mayfield2*, and Julia R.S. Bursten1 1Department of Philosophy, University of Kentucky 2Language Technologies Institute, Carnegie Mellon University christopher.grimsley@uky.edu, elijah@cmu.edu, jrbursten@uky.edu Abstract As the demand for explainable deep learning grows in the evaluation of language technologies, the value of a principled grounding for those explanations grows as well. Here we study the state-of-the-art in explanation for neural models for NLP tasks from the viewpoint of philosophy of science. We focus on recent evaluation work that finds brittleness in explanations obtained through attention mechanisms. We harness philosophical accounts of explanation to suggest broader conclusions from these studies. From this analysis, we assert the impossibility of causal explanations from attention layers over text data. We then introduce NLP researchers to contemporary philosophy of science theories that allow robust yet non-causal reasoning in explanation, giving computer scientists a vocabulary for future research. Keywords: philosophy of science, explainability, causal reasoning, attention mechanisms. * denotes equal contributions. 1. Introduction In the natural language processing community, we’ve reached a consensus that explainability in trained models is a positive attribute. When performing model selection, the less complex and more explainable model should be pre- ferred (holding all else - for instance, classification accuracy or training time - equal). Part of this is purely intuitive and based on logistic ease for software developers; the prefer- ence for explainable models is also spurred on by regula- tion, led by the European Union’s “right to explanation” in the 2016 enactment of the GDPR (Goodman and Flaxman, 2017). Yet so far, most researchers in NLP that describe their models as explainable have treated explanation the way that U.S Supreme Court Justice Potter Stewart famously treated obscenity: “I know it when I see it” (Stewart, concurring, 1964). It is challenging to evaluate the success of an explain- able neural model without defining a criterion for evaluating what counts as an explanation, and moreover what counts as a good explanation. This may be because the phenomenon of giving, receiving, and incorporating explanations is like the breathed air of the process of scientific reasoning: explanation is how knowl- edge is transmitted, how mechanisms and causal processes are learned, how experimental design problems are solved, and how publications and presentations are assembled. Be- cause it appears everywhere, its value and form is assumed. However, like air, explanation can vary dramatically in dif- ferent situations, and assuming that its structure and function in one setting can be automatically exported in another can be as dangerous as mistaking oxygen for nitrogen. Research in philosophy of science has produced decades of results on the nature of scientific explanations across fields, and related those findings to case studies across scientific disciplines. In the NLP community, some of this research has been anticipated by, and recapitulated in, …

Content truncated to 500 words