CREST: A Joint Framework for Rationalization and Counterfactual Text Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15109–15126 July 9-14, 2023 ©2023 Association for Computational Linguistics CREST: A Joint Framework for Rationalization and Counterfactual Text Generation Marcos Treviso1,2∗, Alexis Ross3, Nuno M. Guerreiro1,2, André F. T. Martins1,2,4 1Instituto de Telecomunicações, Lisbon, Portugal 2Instituto Superior Técnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal 3Massachusetts Institute of Technology 4Unbabel, Lisbon, Portugal Abstract Selective rationales and counterfactual exam- ples have emerged as two effective, comple- mentary classes of interpretability methods for analyzing and training NLP models. However, prior work has not explored how these meth- ods can be integrated to combine their comple- mentary advantages. We overcome this limita- tion by introducing CREST (ContRastive Edits with Sparse raTionalization), a joint framework for selective rationalization and counterfactual text generation, and show that this framework leads to improvements in counterfactual quality, model robustness, and interpretability. First, CREST generates valid counterfactuals that are more natural than those produced by pre- vious methods, and subsequently can be used for data augmentation at scale, reducing the need for human-generated examples. Second, we introduce a new loss function that lever- ages CREST counterfactuals to regularize se- lective rationales and show that this regulariza- tion improves both model robustness and ratio- nale quality, compared to methods that do not leverage CREST counterfactuals. Our results demonstrate that CREST successfully bridges the gap between selective rationales and coun- terfactual examples, addressing the limitations of existing methods and providing a more com- prehensive view of a model’s predictions. 1 Introduction As NLP models have become larger and less trans- parent, there has been a growing interest in devel- oping methods for finer-grained interpretation and control of their predictions. One class of meth- ods leverages selective rationalization (Lei et al., 2016; Bastings et al., 2019), which trains models to first select rationales, or subsets of relevant input to- kens, and then make predictions based only on the selected rationales. These methods offer increased interpretability, as well as learning benefits, such ∗Correspondence to: marcos.treviso@tecnico.pt Figure 1: Our generation procedure consists of two stages: (i) a mask stage that highlights relevant tokens in the input through a learnable masker; and (ii) an edit stage, which receives a masked input and uses a masked language model to infill spans conditioned on a prepended label. as improved robustness to input perturbations (Jain et al., 2020; Chen et al., 2022). Another class of methods generates counterfactual examples, or modifications to input examples that change their labels. By providing localized views of decision boundaries, counterfactual examples can be used as explanations of model predictions, contrast datasets for fine-grained evaluation, or new training data- points for learning more robust models (Ross et al., 2021; Gardner et al., 2020; Kaushik et al., 2020). This paper is motivated by the observation that selective rationales and counterfactual examples allow for interpreting and controlling model be- havior through different means: selective rational- ization improves model transparency by weaving interpretability into a model’s internal decision- making …

Content truncated to 500 words