SPECTRA: Sparse Structured Text Rationalization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6534–6550 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6534 SPECTRA: Sparse Structured Text Rationalization Nuno M. Guerreiro1,2 and André F. T. Martins1,2,3 1Instituto de Telecomunicações, Lisbon, Portugal 2Instituto Superior Técnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal 3Unbabel, Lisbon, Portugal {nuno.s.guerreiro, andre.t.martins}@tecnico.ulisboa.pt Abstract Selective rationalization aims to produce de- cisions along with rationales (e.g., text high- lights or word alignments between two sen- tences). Commonly, rationales are modeled as stochastic binary masks, requiring sampling- based gradient estimators, which complicates training and requires careful hyperparameter tuning. Sparse attention mechanisms are a de- terministic alternative, but they lack a way to regularize the rationale extraction (e.g., to con- trol the sparsity of a text highlight or the num- ber of alignments). In this paper, we present a unified framework for deterministic extrac- tion of structured explanations via constrained inference on a factor graph, forming a differen- tiable layer. Our approach greatly eases train- ing and rationale regularization, generally out- performing previous work on what comes to performance and plausibility of the extracted rationales. We further provide a comparative study of stochastic and deterministic methods for rationale extraction for classification and natural language inference tasks, jointly as- sessing their predictive power, quality of the explanations, and model variability. 1 Introduction Selective rationalization (Lei et al., 2016; Bastings et al., 2019; Swanson et al., 2020) is a powerful ex- plainability method, in which we construct models (rationalizers) that produce an explanation or ra- tionale (e.g: text highlights or alignments; Zaidan et al. 2007) along with the decision. One, if not the main, drawback of rationalizers is that it is difficult to train the generator and the predictor jointly under instance-level supervision (Jain et al., 2020). Hard attention mechanisms that stochastically sample rationales employ regular- ization to encourage sparsity and contiguity, and make it necessary to estimate gradients using the score function estimator (SFE), also known as RE- INFORCE (Williams, 1992), or reparameterized gradients (Kingma and Welling, 2014; Jang et al., 2017). Both of these factors substantially compli- cate training by requiring sophisticated hyperpa- rameter tuning and lead to brittle and fragile mod- els that exhibit high variance over multiple runs. Other works use strategies such as top-k to map token-level scores to rationales, but also require gradient estimations to train both modules jointly (Paranjape et al., 2020; Chang et al., 2020). In turn, sparse attention mechanisms (Treviso and Martins, 2020) are deterministic and have exact gradients, but lack a direct way to control sparsity and con- tiguity in the rationale extraction. This raises the question: how can we build an easy-to-train fully differentiable rationalizer that allows for flexible constrained rationale extraction? To answer this question, we introduce sparse structured text rationalization (SPECTRA), which employs LP-SparseMAP (Niculae and Mar- tins, 2020), a constrained structured prediction al- gorithm, to provide a deterministic, flexible and modular rationale extraction process. We exploit our method’s inherent flexibility to extract high- lights and interpretable text matchings …

Content truncated to 500 words