A causal framework for explaining the predictions of black-box sequence-to-sequence models

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412–421 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A causal framework for explaining the predictions of black-box sequence-to-sequence models David Alvarez-Melis and Tommi S. Jaakkola CSAIL, MIT {davidam, tommi}@csail.mit.edu Abstract We interpret the predictions of any black- box structured input-structured output model around a specific input-output pair. Our method returns an “explanation” con- sisting of groups of input-output tokens that are causally related. These dependen- cies are inferred by querying the black-box model with perturbed inputs, generating a graph over tokens from the responses, and solving a partitioning problem to se- lect the most relevant components. We fo- cus the general approach on sequence-to- sequence problems, adopting a variational autoencoder to yield meaningful input per- turbations. We test our method across sev- eral NLP sequence generation tasks. 1 Introduction Interpretability is often the first casualty when adopting complex predictors. This is particularly true for structured prediction methods at the core of many natural language processing tasks such as machine translation (MT). For example, deep learning models for NLP involve a large num- ber of parameters and complex architectures, mak- ing them practically black-box systems. While such systems achieve state-of-the-art results in MT (Bahdanau et al., 2014), summarization (Rush et al., 2015) and speech recognition (Chan et al., 2015), they remain largely uninterpretable, al- though attention mechanisms (Bahdanau et al., 2014) can shed some light on how they operate. Stronger forms of interpretability could offer several advantages, from trust in model predic- tions, error analysis, to model refinement. For example, critical medical decisions are increas- ingly being assisted by complex predictions that should lend themselves to easy verification by hu- man experts. Without understanding how inputs get mapped to the outputs, it is also challenging to diagnose the source of potential errors. A slightly less obvious application concerns model improve- ment (Ribeiro et al., 2016) where interpretability can be used to detect biases in the methods. Interpretability has been approached primarily from two main angles: model interpretability, i.e., making the architecture itself interpretable, and prediction interpretability, i.e., explaining particu- lar predictions of the model (cf. (Lei et al., 2016)). Requiring the model itself to be transparent is of- ten too restrictive and challenging to achieve. In- deed, prediction interpretability can be more eas- ily sought a posteriori for black-box systems in- cluding neural networks. In this work, we propose a novel approach to prediction interpretability with only oracle access to the model generating the prediction. Following (Ribeiro et al., 2016), we turn the local behavior of the model around the given input into an inter- pretable representation of its operation. In con- trast to previous approaches, we consider struc- tured prediction where both inputs and outputs are combinatorial objects, and our explanation con- sists of a summary of operation rather than a sim- pler prediction method. Our method returns an “explanation” consisting of sets of input and output tokens that are causally related under the …

Content truncated to 500 words