Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 486–495, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 486 Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification Vivian Lai and Zheng Cai and Chenhao Tan Department of Computer Science University of Colorado Boulder Boulder, CO vivian.lai, jon.z.cai, chenhao.tan@colorado.edu Abstract Feature importance is commonly used to ex- plain machine predictions. While feature im- portance can be derived from a machine learn- ing model with a variety of methods, the con- sistency of feature importance via different methods remains understudied. In this work, we systematically compare feature importance from built-in mechanisms in a model such as attention values and post-hoc methods that ap- proximate model behavior such as LIME. Us- ing text classification as a testbed, we find that 1) no matter which method we use, impor- tant features from traditional models such as SVM and XGBoost are more similar with each other, than with deep learning models; 2) post- hoc methods tend to generate more similar im- portant features for two models than built-in methods. We further demonstrate how such similarity varies across instances. Notably, im- portant features do not always resemble each other better when two models agree on the pre- dicted label than when they disagree. 1 Introduction As machine learning models are adopted in so- cietally important tasks such as recidivism pre- diction and loan approval, explaining machine predictions has become increasingly important (Doshi-Velez and Kim, 2017; Lipton, 2016). Ex- planations can potentially improve the trustworthi- ness of algorithmic decisions for decision makers, facilitate model developers in debugging, and even allow regulators to identify biased algorithms. A popular approach to explaining machine pre- dictions is to identify important features for a particular prediction (Luong et al., 2015; Ribeiro et al., 2016; Lundberg and Lee, 2017). Typically, these explanations assign a value to each feature (usually a word in NLP), and thus enable visual- izations such as highlighting top k features. In general, there are two classes of methods: 1) built-in feature importance that is embedded in the machine learning model such as coefficients in lin- ear models and attention values in attention mech- anisms; 2) post-hoc feature importance through credit assignment based on the model such as LIME. It is well recognized that robust evalua- tion of feature importance is challenging (Jain and Wallace, 2019; Nguyen, 2018, inter alia), which is further complicated by different use cases of ex- planations (e.g., for decision makers vs. for de- velopers). Throughout this work, we refer to ma- chine learning models that learn from data as mod- els and methods to obtain local explanations (i.e., feature importance in this work) for a prediction by a model as methods. While prior research tends to focus on the inter- nals of models in designing and evaluating meth- ods of explanations, e.g., how well explanations reflect the original model (Ribeiro et …

Content truncated to 500 words