Does Self-Rationalization Improve Robustness to Spurious Correlations?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7403 - 7416 December 7-11, 2022 ©2022 Association for Computational Linguistics Does Self-Rationalization Improve Robustness to Spurious Correlations? Alexis Ross†∗ Matthew E. Peters‡ Ana Marasovi´c§∗ †Massachusetts Institute of Technology, Cambridge, MA, USA ‡Allen Institute for AI, Seattle, WA, USA §University of Utah, Salt Lake City, UT, USA alexisro@mit.edu matthewp@allenai.org ana.marasovic@utah.edu Abstract Rationalization is fundamental to human rea- soning and learning. NLP models trained to produce rationales along with predictions, called self-rationalization models, have been investigated for their interpretability and util- ity to end-users. However, the extent to which training with human-written rationales facili- tates learning remains an under-explored ques- tion. We ask whether training models to self- rationalize can aid in their learning to solve tasks for the right reasons. Specifically, we evaluate how training self-rationalization mod- els with free-text rationales affects robustness to spurious correlations in fine-tuned encoder- decoder and decoder-only models of six dif- ferent sizes. We evaluate robustness to spu- rious correlations by measuring performance on 1) manually annotated challenge datasets and 2) subsets of original test sets where re- liance on spurious correlations would fail to produce correct answers. We find that while self-rationalization can improve robustness to spurious correlations in low-resource settings, it tends to hurt robustness in higher-resource settings. Furthermore, these effects depend on model family and size, as well as on ratio- nale content. Together, our results suggest that explainability can come at the cost of robust- ness; thus, appropriate care should be taken when training self-rationalizing models with the goal of creating more trustworthy models. 1 Introduction Rationalization—the process of explaining the rea- soning used to come to a particular decision—plays a pivotal role in human inference and learning (Lombrozo, 2016). For these reasons, there has been a growing interest in producing NLP mod- *Work undertaken while Alexis Ross and Ana Marasovi´c were at the Allen Institute for AI. Our code is publicly available at https://github.com/ allenai/rationale_robustness els that can output rationales1 for their predic- tions. Models that output such rationales have mul- tiple benefits: First, they are more interpretable and easier to interact with for end-users than non- rationalizing models (Alvarez-Melis and Jaakkola, 2018). Second, such intermediate rationalization can offer learning benefits, such as achieving com- parable performance with less data and improving out-of-distribution generalization (Nye et al., 2021; Wei et al., 2022; Zelikman et al., 2022). However, the question of whether training mod- els to rationalize can help them learn how to solve tasks for the right reasons remains open. In par- ticular, rationales encode information about the underlying reasoning humans use to reach answers, which raises the question: Does incorporating such rationales into training allow models to rely on human-aligned reasoning rather than spurious fea- ture interactions? If so, training with rationales could offer a pathway towards creating more ro- bust, trustworthy, or cognitively plausible models. In this work, we explore this question by empir- ically investigating whether training models with human-written rationales can help make …

Content truncated to 500 words