Evaluating and Characterizing Human Rationales

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9294–9307, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9294 Evaluating and Characterizing Human Rationales Samuel Carton∗, Anirudh Rathore∗, Chenhao Tan University of Colorado Boulder {samuel.carton,anirudh.rathore,chenhao.tan}@colorado.edu Abstract Two main approaches for evaluating the qual- ity of machine-generated rationales are: 1) us- ing human rationales as a gold standard; and 2) automated metrics based on how rationales affect model behavior. An open question, how- ever, is how human rationales fare with these automatic metrics. Analyzing a variety of datasets and models, we find that human ra- tionales do not necessarily perform well on these metrics. To unpack this finding, we pro- pose improved metrics to account for model- dependent baseline performance. We then pro- pose two methods to further characterize ra- tionale quality, one based on model retraining and one on using “fidelity curves” to reveal properties such as irrelevance and redundancy. Our work leads to actionable suggestions for evaluating and characterizing rationales. 1 Introduction Explanations in NLP often take the form of ra- tionales, subsets of input tokens that are consid- ered important to the model’s decision (DeYoung et al., 2020). As interest in explainable AI has in- creased, so has interest in evaluating the quality of explanatory rationales. However, this is a challeng- ing task because it can be difficult to pin down ex- actly what constitutes “good” rationales for model predictions (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019). Two main strategies that have been proposed in recent work are: 1) to view human-generated rationales as a gold standard and evaluate model- generated rationales in comparison to them; and 2) to assess the “fidelity” of a rationale to a prediction using automatic metrics. The human-gold-standard approach views ratio- nales as an additional form of label that can be col- lected alongside document-level labels. Because ∗Equal contribution. NLP tasks tend to involve human-generated labels, it makes intuitive sense that human-generated ra- tionales might be considered authoritative. When human rationales are not available, eval- uations of machine rationales turn to automatic metrics. These metrics divorce rationale evaluation from an external standard, seeking instead to judge whether rationales are coherent relative to model behavior. Popular recent metrics are sufficiency and comprehensiveness (i.e., necessity), which as- sess whether a rationale is sufficient/necessary for a model prediction by comparing the model’s behav- ior on the full input to its behavior on input masked according to the rationale or its complement. We use the term fidelity to refer jointly to sufficiency and comprehensiveness. To the best of our knowledge, no existing work has systematically examined human rationales us- ing these automatic metrics. However, this is an important step towards evaluating rationales be- cause it helps characterize the disparities between the two types of approach. Are human rationales sufficient to allow models to predict human labels? Are they comprehensive? And what other insights can we gain about human rationales and fidelity metrics by performing this assessment? In practice, …

Content truncated to 500 words