Measuring and Improving Faithfulness of Attention in Neural Machine Translation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2791–2802 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2791 Measuring and Improving Faithfulness of Attention in Neural Machine Translation Pooya Moradi, Nishant Kambhatla, Anoop Sarkar School of Computing Science Simon Fraser University Burnaby, BC, Canada {pmoradi, nkambhat, anoop}@sfu.ca Abstract While the attention heatmaps produced by neu- ral machine translation (NMT) models seem insightful, there is little evidence that they re- flect a model’s true internal reasoning. We provide a measure of faithfulness for NMT based on a variety of stress tests where atten- tion weights which are crucial for prediction are perturbed and the model should alter its predictions if the learned weights are a faithful explanation of the predictions. We show that our proposed faithfulness measure for NMT models can be improved using a novel differen- tiable objective that rewards faithful behaviour by the model through probability divergence. Our experimental results on multiple language pairs show that our objective function is effec- tive in increasing faithfulness and can lead to a useful analysis of NMT model behaviour and more trustworthy attention heatmaps. Our pro- posed objective improves faithfulness without reducing the translation quality and has a use- ful regularization effect on the NMT model and can even improve translation quality in some cases. 1 Introduction How trustworthy are our neural models? This ques- tion has led to a wide variety of contemporary NLP research focusing on (a) different axes of inter- pretability including plausibility (or interchange- ably human-interpretability) (Herman, 2017; Lage et al., 2019) and faithfulness (Lipton, 2018; Jacovi and Goldberg, 2020b), (b) interpretation of the neu- ral model components (Belinkov et al., 2017; Dalvi et al., 2017; Vig and Belinkov, 2019), (c) explain- ing the decisions made by neural models to hu- mans (using explanations, highlights, rationales, etc.) (Ribeiro et al., 2016; Li et al., 2016; Ding et al., 2017; Ghaeini et al., 2018; Bastings et al., 2019; Jain et al., 2020), and (d) evaluating different explanation methods from different perspectives je to moorův zákon za posledních sto let je to moorův zákon za posledních sto let it’s moore’s law for the last century it’s moore’s law for the last century 0.00 1.00 attention weights Figure 1: An example of unfaithful attention weights produced during a Cs-En translation. Note in the left at- tention heatmap, the attention is on the word sto while the decoder generates century. However, in the right heatmap, sto is not attended to at all but century is still produced as the output. This shows unfaithful behavior. (Samek et al., 2016; Mohseni and Ragan, 2018; Po- erner et al., 2018; Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Li et al., 2020). All of these approaches make NLP neural mod- els more trustworthy. In this work, we focus on faithfulness which intuitively provides the extent to which an explanation accurately represents the true reasoning behind a prediction. It is particularly important …

Content truncated to 500 words