A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1556 - 1576 December 7-11, 2022 ©2022 Association for Computational Linguistics A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference Kerem Zaman∗ UNC Chapel Hill kzaman@cs.unc.edu Yonatan Belinkov† Technion – Israel Institute of Technology belinkov@technion.ac.il Abstract Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility. First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations. We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods. Finally, we augment the XNLI dataset with highlight-based explanations, providing a mul- tilingual NLI dataset with highlights, to sup- port future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.1 1 Introduction The opaqueness of large pre-trained models like BERT (Devlin et al., 2019) and GPT (Radford and Narasimhan, 2018) motivates developing ex- planation methods (Wallace et al., 2020), which aim to attribute importance to particular input fea- tures (Springenberg et al., 2015; Bach et al., 2015; Ribeiro et al., 2016; Sundararajan et al., 2017), such as words in a textual input. Two main crite- ria for evaluating such methods are plausibility and faithfulness (Jacovi and Goldberg, 2020). Plausibil- ity can be defined as the consistency between expla- nations and human expectations, while faithfulness is defined as the consistency between explanations and the model’s underlying decision-making pro- cess. Prior evaluations of attributions along these di- mensions (Atanasova et al., 2020; DeYoung et al., ∗Work done while at Bo˘gaziçi University. † Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion. 1Our code and data are available at https://www. keremzaman.com/explaiNLI. 2020; Ding and Koehn, 2021) suffer from several limitations. First, they have been limited in the range of considered attribution methods and the mechanism of calculating the attributions. Second, standard faithfulness evaluations, such as erasure- based ones (DeYoung et al., 2020), entail running the model on examples outside of the training distri- bution (Bastings and Filippova, 2020). Third, prior plausibility evaluations are limited to English-only datasets due to the lack of multilingual datasets with highlighted explanations. In this work, we aim to fill these gaps. Our main contribution is a new framework for evaluating the faithfulness of attribution methods. Inspired by Jacovi and Goldberg (2020)’s criterion for faith- ful explanations as giving similar explanations for similar inputs, we propose to use cross-lingual sen- tences (translations) as similar inputs. Given a mul- tilingual model, we argue that faithful attributions should point to words that are aligned in two trans- lations of the same sentence. This approach avoids out-of-distribution inputs by utilizing cross-lingual sentences as naturally ocurring input perturbations. We focus on Natural Language Inference (NLI) as a case study, since it is a central task that …

Content truncated to 500 words