An Empirical Study on Explanations in Out-of-Domain Settings

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6920 - 6938 May 22-27, 2022 c⃝2022 Association for Computational Linguistics An Empirical Study on Explanations in Out-of-Domain Settings George Chrysostomou Nikolaos Aletras Department of Computer Science, University of Sheffield United Kingdom {gchrysostomou1, n.aletras}@sheffield.ac.uk Abstract Recent work in Natural Language Processing has focused on developing approaches that ex- tract faithful explanations, either via identify- ing the most important tokens in the input (i.e. post-hoc explanations) or by designing inher- ently faithful models that first select the most important tokens and then use them to predict the correct label (i.e. select-then-predict mod- els). Currently, these approaches are largely evaluated on in-domain settings. Yet, little is known about how post-hoc explanations and inherently faithful models perform in out-of- domain settings. In this paper, we conduct an extensive empirical study that examines: (1) the out-of-domain faithfulness of post-hoc ex- planations, generated by five feature attribu- tion methods; and (2) the out-of-domain perfor- mance of two inherently faithful models over six datasets. Contrary to our expectations, re- sults show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain. We find this mislead- ing and suggest using a random baseline as a yardstick for evaluating post-hoc explanation faithfulness. Our findings also show that select- then predict models demonstrate comparable predictive performance in out-of-domain set- tings to full-text trained models.1 1 Introduction An explanation or rationale2, typically consists of a subset of the input that contributes more to the pre- diction. Extracting faithful explanations is impor- tant for studying model behavior (Adebayo et al., 2020) and assisting in tasks requiring human de- cision making, such as clinical text classification (Chakrabarty et al., 2019), misinformation detec- tion (Popat et al., 2018; Mu and Aletras, 2020) and legal text classification (Chalkidis et al., 2019, 1Code available at: https://github.com/ GChrysostomou/ood_faith 2We use these terms interchangeably throughout our work. 2021). A faithful explanation is one which accu- rately represents the reasoning behind a model’s prediction (Jacovi and Goldberg, 2020) Two popular methods for extracting explanations are through feature attribution approaches (i.e. post- hoc explanation methods) or via inherently faithful classifiers (i.e. select-then-predict models). The first computes the contribution of different parts of the input with respect to a model’s prediction (Sundararajan et al., 2017; Ribeiro et al., 2016; Shrikumar et al., 2017). The latter consists of using a rationale extractor to identify the most important parts of the input and a rationale classifier, a model trained using as input only the extractor’s rationales (Bastings et al., 2019; Jain et al., 2020; Guerreiro and Martins, 2021).3 Figure 1 illustrates the two approaches with an example. Currently, these explanation methods have been mostly evaluated on in-domain settings (i.e. the train and test data come from the same distribution). However, when deploying models in real-world ap- plications, inference might be performed on data from a different distribution, i.e. out-of-domain (Desai and Durrett, 2020; Ovadia et al., 2019). …

Content truncated to 500 words