Rule induction for global explanation of trained models

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 82–97 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 82 Rule induction for global explanation of trained models Madhumita Sushil, Simon ˇSuster and Walter Daelemans Computational Linguistics & Psycholinguistics Research Center, University of Antwerp, Belgium firstname.lastname@uantwerpen.be Abstract Understanding the behavior of a trained net- work and finding explanations for its outputs is important for improving the network’s per- formance and generalization ability, and for ensuring trust in automated systems. Several approaches have previously been proposed to identify and visualize the most important fea- tures by analyzing a trained network. How- ever, the relations between different features and classes are lost in most cases. We pro- pose a technique to induce sets of if-then-else rules that capture these relations to globally explain the predictions of a network. We first calculate the importance of the features in the trained network. We then weigh the original inputs with these feature importance scores, simplify the transformed input space, and fi- nally fit a rule induction model to explain the model predictions. We find that the output rule-sets can explain the predictions of a neu- ral network trained for 4-class text classifi- cation from the 20 newsgroups dataset to a macro-averaged F-score of 0.80. We make the code available at https://github.com/ clips/interpret_with_rules. 1 Introduction Deep, non-linear neural networks are notorious for being black boxes, because the basis of a net- work’s decision is unknown. Although some- times we only care about better performance, un- derstanding a trained model is important in many cases. For example, when a statistical system is used to take decisions regarding a patient’s health, it is critical to know the underlying reasons. Caru- ana et al. (2015) have previously discussed a rule- based system that had associated the history of asthma in patients suffering from pneumonia with a lower risk of death due to it. Despite being coun- terintuitive, it was a predictive pattern in the data because the patients with asthma were admitted di- rectly to the ICU and received more intensive care, which resulted in better outcomes. Model inter- pretability is also useful to understand the biases in the data that influence its decision. For exam- ple, explaining a trained model and its outputs can bring attention towards a potentially unfair out- come when a loan or a job opportunity is denied to an individual due to any societal bias present in the training data1,2. Another less discussed aspect of model interpretability is its utility for analyz- ing a model’s strengths and weaknesses. This un- derstanding can assist with improving the model’s performance and generalization ability (Andrews et al., 1995). Model interpretability techniques can either have a global or a local scope. A global explana- tion refers to the explanation of a complete model, as opposed to local explanations of individual pre- dictions. Several existing model-agnostic inter- pretability techniques provide a list of important features as explanations. In such a list, the …

Content truncated to 500 words