How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3243–3255, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3243 How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking Nicola De Cao 1,2, Michael Schlichtkrull 1,2, Wilker Aziz 1, Ivan Titov 1,2 1University of Amsterdam, 2University of Edinburgh { nicola.decao, m.s.schlichtkrull, w.aziz } @uva.nl ititov@inf.ed.ac.uk Abstract Attribution methods assess the contribution of inputs to the model prediction. One way to do so is erasure: a subset of inputs is considered irrelevant if it can be removed without affect- ing the prediction. Though conceptually sim- ple, erasure’s objective is intractable and ap- proximate search remains expensive with mod- ern deep NLP models. Erasure is also suscepti- ble to the hindsight bias: the fact that an input can be dropped does not mean that the model ‘knows’ it can be dropped. The resulting prun- ing is over-aggressive and does not reflect how the model arrives at the prediction. To deal with these challenges, we introduce Differen- tiable Masking. DIFFMASK learns to mask- out subsets of the input while maintaining dif- ferentiability. The decision to include or dis- regard an input token is made with a simple model based on intermediate hidden layers of the analyzed model. First, this makes the ap- proach efficient because we predict rather than search. Second, as with probing classifiers, this reveals what the network ‘knows’ at the corresponding layers. This lets us not only plot attribution heatmaps but also analyze how de- cisions are formed across network layers. We use DIFFMASK to study BERT models on sen- timent classification and question answering.1 1 Introduction Deep neural networks have become standard tools in NLP demonstrating impressive improvements over traditional approaches on many tasks (Gold- berg, 2017). Their power typically comes at the ex- pense of interpretability, which may prevent users from trusting predictions (Kim, 2015; Ribeiro et al., 2016), makes it hard to detect model or data de- ficiencies (Gururangan et al., 2018; Kaushik and 1Source code available at https://github.com/ nicola-decao/diffmask Gated input Model Model with gated input Figure 1: DIFFMASK: hidden states up to layer ℓ from a model (top) are fed to a classifier g that predicts a mask z. We use this to mask the input and re-compute the forward pass (bottom). The classifier g is trained to mask the input as much as possible without changing the output (minimizing a divergence D⋆). Lipton, 2018) or verify that a model is fair and does not exhibit harmful biases (Sun et al., 2019; Holstein et al., 2019). These challenges have motivated work on inter- pretability, both in NLP and generally in machine learning; see Belinkov and Glass (2019) and Jacovi and Goldberg (2020) for reviews. In this work, we study post hoc interpretability where the goal is to explain the prediction of a trained model and to reveal how the model arrives at the decision. This goal is usually approached with attribution meth- ods (Bach et al., 2015; Shrikumar …

Content truncated to 500 words