An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1938–1952, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1938 An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction Bhargavi Paranjape† Mandar Joshi† John Thickstun† Hannaneh Hajishirzi†ϵ Luke Zettlemoyer† † Allen School of Computer Science & Engineering, University of Washington, Seattle, WA ϵAllen Institute of Artificial Intelligence, Seattle {bparan,mandar90,thickstn,hannaneh,lsz}@cs.washington.edu Abstract Decisions of complex models for language un- derstanding can be explained by limiting the inputs they are provided to a relevant sub- sequence of the original text — a rationale. Models that condition predictions on a con- cise rationale, while being more interpretable, tend to be less accurate than models that are able to use the entire context. In this paper, we show that it is possible to better manage the trade-off between concise explanations and high task accuracy by optimizing a bound on the Information Bottleneck (IB) objective. Our approach jointly learns an explainer that pre- dicts sparse binary masks over input sentences without explicit supervision, and an end-task predictor that considers only the residual sen- tences. Using IB, we derive a learning objec- tive that allows direct control of mask spar- sity levels through a tunable sparse prior. Ex- periments on the ERASER benchmark demon- strate significant gains over previous work for both task performance and agreement with hu- man rationales. Furthermore, we find that in the semi-supervised setting, a modest amount of gold rationales (25% of training examples with gold masks) can close the performance gap with a model that uses the full input.1 1 Introduction A rationale is a short yet sufficient part of the input text that can explain model decisions for a range of language understanding tasks (Lei et al., 2016). Models can be faithful to a rationale by only us- ing the selected text as input for end-task predic- tion (DeYoung et al., 2019). However, there is almost always a trade-off between interpretable models that learn to extract sparse rationales and more accurate models that are able to use the full 1Our code is available at https://github.com/ bhargaviparanjape/explainable_qa Text Homo�sapiens�is�the�binomial�nomenclature for�the�only�extant�human�species�.� Homo�is�the�human�genus�,�which�also�includes Neanderthals�…� Modern�humans�are�the�subspecies�Homo sapiens�…. The�ingenuity�and�adaptability�of�Homo�sapiens …� It�is�currently�of�least�concern�on�the�Red List�of�endangered�species�by�the�IUCN Homo�sapiens�is�the�binomial�nomen- clature�for�the�only�extant�human�species� Rationale Controlled�Sparsity�=�40% Q:�Homo�sapiens�are�on�the�IUCN�Red�List. Boolean mask Input Sentences Refutes Label It�is�currently�of�least�concern�on�the�Red List�of�endangered�species�by�the�IUCN Figure 1: Our Information Bottleneck-based approach extracts concise rationales that are minimally informa- tive about the original input, and maximally informa- tive about the label through fine-grained control of spar- sity in the bottleneck (0.4 in this fact verification exam- ple). End-task prediction is conditioned only on the bottlenecked input. context but provide little explanation for their pre- dictions (Lei et al., 2016; Weld and Bansal, 2019). In this paper, we show that it is possible to better manage this trade-off by optimizing a novel bound on the Information Bottleneck (Tishby et al., 1999) objective (Figure 1). We follow recent work in representing rationales as binary masks over the input text (Lei et al., 2016; Bastings et al., 2019). During learning, it is com- mon to encourage …

Content truncated to 500 words