Mitigating the Inconsistency Between Word Saliency and Model Confidence with Pathological Contrastive Training

Findings of the Association for Computational Linguistics: ACL 2022, pages 2226 - 2244 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Mitigating the Inconsistency Between Word Saliency and Model Confidence with Pathological Contrastive Training Pengwei Zhan§‡, Yang Wu§‡, Shaolei Zhou§‡, Yunjian Zhang§‡, Liming Wang§∗ §Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China ‡School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China {zhanpengwei,wuyang0419}@iie.ac.cn {zhoushaolei,zhangyunjian,wangliming}@iie.ac.cn Abstract Neural networks are widely used in various NLP tasks for their remarkable performance. However, the complexity makes them difficult to interpret, i.e., they are not guaranteed right for the right reason. Besides the complexity, we reveal that the model pathology - the in- consistency between word saliency and model confidence, further hurts the interpretability. We show that the pathological inconsistency is caused by the representation collapse is- sue, which means that the representation of the sentences with tokens in different saliency reduced is somehow collapsed, and thus the important words cannot be distinguished from unimportant words in terms of model confi- dence changing. In this paper, to mitigate the pathology and obtain more interpretable models, we propose Pathological Contrastive Training (PCT) framework, which adopts con- trastive learning and saliency-based samples augmentation to calibrate the sentences repre- sentation. Combined with qualitative analysis, we also conduct extensive quantitative exper- iments and measure the interpretability with eight reasonable metrics. Experiments show that our method can mitigate the model pathol- ogy and generate more interpretable models while keeping the model performance. Abla- tion study also shows the effectiveness. 1 Introduction Neural networks have achieved remarkable success in various NLP tasks, while the extremely high complexity of such models makes them difficult to interpret. Complex models may learn significantly different attributions with similar accuracy during training as datasets are often full of ambiguities (Ross et al., 2017). If a model is deployed without ensuring that it is right for the right reason, it may completely fail to make reliable predictions on new data, which is very dangerous. For example, some ∗Corresponding Author. This is a wonderful movie full of grace and hope. This is a wonderful movie full of grace and hope. This is a wonderful movie full of grace and hope. reduce unimportant words reduce important words Normal PCT Confidence Confidence Confidence Confidence Confidence Confidence Positive Negative Original result Saliency Figure 1: Word saliency and model confidence on case sentence. Normal model can not distinguish well be- tween the influence of important and unimportant words, and the confidence on Positive class always focuses on a high region. Our method mitigates the pathology. models will counter-intuitively consider preposi- tions to have extremely high saliency in rumor de- tection tasks. Interpretable models can ensure that the attribution of model prediction is consistent with human intuition, allowing the model to be trusted in critical applications. In addition to the complexity, the pathology also makes models more difficult to interpret (Feng et al., 2018). Neural networks are more linear than expected, leading models to overfit the negative log- likelihood …

Content truncated to 500 words