Learning to Deceive with Attention-Based Explanations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782–4793 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4782 Learning to Deceive with Attention-Based Explanations Danish Pruthi†, Mansi Gupta‡, Bhuwan Dhingra†, Graham Neubig†, Zachary C. Lipton† †Carnegie Mellon University, Pittsburgh, USA ‡Twitter, New York, USA ddanish@cs.cmu.edu, mansig@twitter.com, {bdhingra, gneubig, zlipton}@cs.cmu.edu Abstract Attention mechanisms are ubiquitous compo- nents in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive at- tention masks. Our method diminishes the to- tal weight assigned to designated impermis- sible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explana- tions deceive people into thinking that predic- tions from a model biased against gender mi- norities do not rely on the gender. Conse- quently, our results cast doubt on attention’s reliability as a tool for auditing algorithms in the context of fairness and accountability.1 1 Introduction Since their introduction as a method for aligning inputs and outputs in neural machine translation, attention mechanisms (Bahdanau et al., 2014) have emerged as effective components in various neural network architectures. Attention works by aggregating a set of tokens via a weighted sum, where the attention weights are calculated as a function of both the input encodings and the state of the decoder. Because attention mechanisms allocate weight among the encoded tokens, these coefficients are 1The code and the datasets used in paper are available at https://github.com/danishpruthi/ deceptive-attention Attention Biography Label Original Ms. X practices medicine in Memphis, TN and ... Ms. X speaks English and Spanish. Physician Ours Ms. X practices medicine in Memphis , TN and ... Ms. X speaks English and Spanish. Physician Table 1: Example of an occupation prediction task where attention-based explanation (highlighted) has been manipulated to whitewash problematic tokens. sometimes thought of intuitively as indicating which tokens the model focuses on when making a particular prediction. Based on this loose intu- ition, attention weights are often claimed to ex- plain a model’s predictions. For example, a recent survey on attention (Galassi et al., 2019) remarks: “By inspecting the networks attention, ... one could attempt to investigate and understand the outcome of neural net- works. Hence, weight visualization is now common practice.” In another work, De-Arteaga et al. (2019) study gender bias in machine learning models for occu- pation classification. As machine learning is in- creasingly used in hiring processes for tasks in- cluding resume filtering, the potential for bias raises the spectre that automating this process could lead to social harms. …

Content truncated to 500 words