Prediction of People’s Emotional Response towards Multi-modal News

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 12–31 November 20–23, 2022. ©2022 Association for Computational Linguistics 12 Double Trouble: How to not Explain a Text Classifier’s Decisions Using Counterfactuals Synthesized by Masked Language Models? Thang M. Pham† thangpham@auburn.edu Trung Bui∗ bui@adobe.com Long Mai∗ mai.t.long88@gmail.com Anh Nguyen† anh.ng8@gmail.com †Auburn University ∗Adobe Research Abstract A principle behind dozens of attribution meth- ods is to take the prediction difference between before-and-after an input feature (here, a token) is removed as its attribution. A popular Input Marginalization (IM) method (Kim et al., 2020) uses BERT to replace a token, yielding more plausible counterfactuals. While Kim et al. (2020) reported that IM is effective, we find this conclusion not convincing as the DeletionBERT metric used in their paper is biased towards IM. Importantly, this bias exists in Deletion-based metrics, including Insertion, Sufficiency, and Comprehensiveness. Furthermore, our rigorous evaluation using 6 metrics and 3 datasets finds no evidence that IM is better than a Leave- One-Out (LOO) baseline. We find two reasons why IM is not better than LOO: (1) deleting a single word from the input only marginally reduces a classifier’s accuracy; and (2) a highly predictable word is always given near-zero at- tribution, regardless of its true importance to the classifier. In contrast, making Local Inter- pretable Model-Agnostic Explanations (LIME) counterfactuals more natural via BERT consis- tently improves LIME accuracy under several RemOve-And-Retrain (ROAR) metrics. 1 Introduction Feature attribution maps (AMs), i.e. highlights in- dicating the importance of each input token w.r.t. a classifier’s decision, can help improve human accu- racy on downstream tasks including detecting fake movie reviews (Lai and Tan, 2019) or identifying biases in text classifiers (Liu and Avci, 2019). Many Leave-One-Out (LOO) methods compute the attribution of an input token by measuring the prediction changes after substituting that token’s embedding with zeros (Li et al., 2016; Jin et al., 2020) or [UNK] (Kim et al., 2020). That is, delet- ing or replacing features is the underlying principle of at least 25 attribution methods (Covert et al., 2020). (a) SST – Groundtruth & target class: “positive” S The very definition of the ‘ small ’ movie , but it is a good stepping stone for director Sprecher . 0.9793 stepping 0.9760 stone 0.8712 for 0.0050 rolling 0.0048 stones 0.0860 to 0.0021 casting 0.0043 point 0.0059 , IM0 The very definition of the ‘ small ’ movie , but it is a good stepping stone for director Sprecher . IM1 The very definition of the ‘ small ’ movie , but it is a good stepping stone for director Sprecher . IM2 The very definition of the ‘ small ’ movie , but it is a good stepping stone for director Sprecher . IM3 The very definition of the ‘ small ’ movie , but it is a good stepping stone for director Sprecher . P(b) e-SNLI – Groundtruth …

Content truncated to 500 words