Rationales for Sequential Predictions

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10314–10332 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10314 Rationales for Sequential Predictions Keyon Vafa Columbia University keyon.vafa@columbia.edu Yuntian Deng Harvard University dengyuntian@seas.harvard.edu David M. Blei Columbia University david.blei@columbia.edu Alexander M. Rush Cornell Tech arush@cornell.edu Abstract Sequence models are a critical component of modern NLP systems, but their predictions are difficult to explain. We consider model ex- planations though rationales, subsets of con- text that can explain individual model predic- tions. We find sequential rationales by solving a combinatorial optimization: the best ratio- nale is the smallest subset of input tokens that would predict the same output as the full se- quence. Enumerating all subsets is intractable, so we propose an efficient greedy algorithm to approximate this objective. The algorithm, which is called greedy rationalization, applies to any model. For this approach to be effec- tive, the model should form compatible condi- tional distributions when making predictions on incomplete subsets of the context. This condition can be enforced with a short fine- tuning step. We study greedy rationalization on language modeling and machine translation. Compared to existing baselines, greedy ratio- nalization is best at optimizing the sequential objective and provides the most faithful ratio- nales. On a new dataset of annotated sequen- tial rationales, greedy rationales are most simi- lar to human rationales. 1 Introduction Sequence models are a critical component of gen- eration tasks ranging from language modeling to machine translation to summarization. These tasks are dominated by complex neural networks. While these models produce accurate predictions, their decision making processes are hard to explain. In- terpreting a model’s prediction is important in a va- riety of settings: a researcher needs to understand a model to debug it; a doctor using a diagnostic model requires justifications to validate a decision; a company deploying a language model relies on model explanations to detect biases appropriated from training data. Interpretation takes many flavors (Lipton, 2018). We focus on rationales, i.e. identifying the most The Supreme Court on Tuesday rejected a challenge to the constitutionality of the death penalty The Supreme Court on Tuesday rejected a challenge to the constitutionality of the death penalty Figure 1. Rationales for sequential prediction on GPT- 2. Each row is a predicted word. The dark cells cor- respond to the context words found by greedy rational- ization. To predict “constitutionality”, the model only needs “Supreme”, “Court”, “challenge”, and “the”. important subset of input tokens that leads to the model’s prediction. For example, consider the sen- tence: “The Supreme Court on Tuesday rejected a challenge to the constitutionality of the death penalty.” Suppose we would like to explain the de- cision of the model to generate “constitutionality”. While the model mathematically conditions on all the previous words, only some are necessary for its predictions. In this case, the rationale produced by our algorithm includes “the”, “challenge”, and notably “Supreme Court”, but not phrases that add no information like “on Tuesday” (Figure 1). Various …

Content truncated to 500 words