Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4198 Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? Alon Jacovi Bar Ilan University alonjacovi@gmail.com Yoav Goldberg Bar Ilan University and Allen Institute for AI yoav.goldberg@gmail.com Abstract With the growing popularity of deep-learning based NLP models, comes a need for inter- pretable systems. But what is interpretability, and what constitutes a high-quality interpreta- tion? In this opinion piece we reflect on the current state of interpretability evaluation re- search. We call for more clearly differentiat- ing between different desired criteria an inter- pretation should satisfy, and focus on the faith- fulness criteria. We survey the literature with respect to faithfulness evaluation, and arrange the current approaches around three assump- tions, providing an explicit form to how faith- fulness is “defined” by the community. We provide concrete guidelines on how evaluation of interpretation methods should and should not be conducted. Finally, we claim that the current binary definition for faithfulness sets a potentially unrealistic bar for being considered faithful. We call for discarding the binary no- tion of faithfulness in favor of a more graded one, which we believe will be of greater prac- tical utility. 1 Introduction Fueled by recent advances in deep-learning and language processing, NLP systems are increasingly being used for prediction and decision-making in many fields (Vig and Belinkov, 2019), including sensitive ones such as health, commerce and law (Fort and Couillault, 2016). Unfortunately, these highly flexible and highly effective neural models are also opaque. There is therefore a critical need for explaining learning-based models’ decisions. The emerging research topic of interpretability or explainability1 has grown rapidly in recent years. Unfortunately, not without growing pains. 1Despite fine-grained distinctions between the terms, within the scope of this work we use the terms “interpretability” and “explainability” interchangeably. One such pain is the challenge of defining—and evaluating—what constitutes a quality interpreta- tion. Current approaches define interpretation in a rather ad-hoc manner, motivated by practical use- cases and applications. However, this view often fails to distinguish between distinct aspects of the interpretation’s quality, such as readability, plausi- bility and faithfulness (Herman, 2017).2 We argue (§2, §5) such conflation is harmful, and that faith- fulness should be defined and evaluated explicitly, and independently from plausibility. Our main focus is the evaluation of the faithful- ness of an explanation: a faithful interpretation is one that accurately represents the reasoning pro- cess behind the model’s prediction. We find this to be a pressing issue: in cases where an explanation is required to be faithful, imperfect or misleading evaluation can have disastrous effects. While literature in this area may implicitly or explicitly evaluate faithfulness for specific expla- nation techniques, there is no consistent and for- mal definition of faithfulness. We uncover three assumptions that underlie all these attempts. By making the assumptions explicit and organizing the literature around them, we “connect the dots” be- tween seemingly distinct …

Content truncated to 500 words