MPII: Multi-Level Mutual Promotion for Inference and Interpretation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7074 - 7084 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MPII: Multi-Level Mutual Promotion for Inference and Interpretation Yan Liu1∗, Sanyuan Chen2,3, Yazheng Yang1, Qi Dai3 1School of Computer Science and Technology, Zhejiang University, China 2School of Computer Science and Technology, Harbin Institute of Technology, China 3Microsoft Research, Beijing, China runningmelles@gmail.com, sychen@ir.hit.edu.cn, yazheng_yang@zju.edu.cn, qid@microsoft.com Abstract In order to better understand the rationale be- hind model behavior, recent works have ex- ploited providing interpretation to support the inference prediction. However, existing meth- ods tend to provide human-unfriendly inter- pretation, and are prone to sub-optimal per- formance due to one-side promotion, i.e. ei- ther inference promotion with interpretation or vice versa. In this paper, we propose a multi-level Mutual Promotion mechanism for self-evolved Inference and sentence-level Interpretation (MPII). Specifically, from the model-level, we propose a Step-wise Integra- tion Mechanism to jointly perform and deeply integrate inference and interpretation in an au- toregressive manner. From the optimization- level, we propose an Adversarial Fidelity Reg- ularization to improve the fidelity between in- ference and interpretation with the Adversarial Mutual Information training strategy. Exten- sive experiments on NLI and CQA tasks reveal that the proposed MPII approach can signifi- cantly outperform baseline models for both the inference performance and the interpretation quality.1 1 Introduction Recently, the interpretability of neural networks has been of increasing concern. In order to break the black-box of neural networks, many works explore the interpretability of neural networks through pro- viding interpretations to support their inference re- sults (Ribeiro et al., 2016; Chen et al., 2018; Liu et al., 2019; Thorne et al., 2019; Kumar and Taluk- dar, 2020). Although prior works have made some progress towards interpretable NLP, they tend to provide interpretations that lack human-readability. Ex- isting interpretable models usually extract promi- ∗Work was done during internship at Microsoft Research 1Our code is available at https://github.com/ theNamek/MPII.git Premise: People walk through a store. Hypothesis: Passengers in a car driving down the street. Label: Contradiction Heatmap Explanation Alignment Explanation/Rationale Sentence-level Explanation: People cannot walk while driving down the street. People walk through a store . Passengers a in car driving down the street . Token-level Explanation: Passengers, through, walk, down, car Figure 1: Comparison of different interpretations: heatmap explanation, alignment rationale, token-level NL explanation, and sentence-level NL explanation. nent features or select input key words as expla- nations, such as attention distribution (Xu et al., 2015), heatmap (Samek et al., 2017), alignment rationale (Jiang et al., 2021), gradients (Li et al., 2016), magnitude of hidden states (Linzen et al., 2016), etc. Considering readability and comprehen- sibility for humans, some works turn to generate token-level explanations (Liu et al., 2019; Thorne et al., 2019), which are nevertheless prone to cause ambiguity. Figure 1 shows some prevalent forms of interpretations in NLI task. Obviously, human lan- guage interpretations seem more acceptable than those chaotic maps, whether it is heatmap or align- ment map. As …

Content truncated to 500 words