FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3712–3727 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3712 FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation Kushal Lakhotia†∗ Bhargavi Paranjape‡ Asish Ghoshal† Wen-tau Yih† Yashar Mehdad† Srinivasan Iyer†∗ †Facebook AI ‡University of Washington {kushall, aghoshal, scottyih, mehdad, sviyer}@fb.com bparan@cs.washington.edu Abstract Natural language (NL) explanations of model predictions are gaining popularity as a means to understand and verify decisions made by large black-box pre-trained models, for tasks such as Question Answering (QA) and Fact Verification. Recently, pre-trained sequence to sequence (seq2seq) models have proven to be very effective in jointly making predictions, as well as generating NL explanations. However, these models have many shortcomings; they can fabricate explanations even for incorrect predictions, they are difficult to adapt to long input documents, and their training requires a large amount of labeled data. In this paper, we develop FiD-Ex1, which addresses these shortcomings for seq2seq models by: 1) intro- ducing sentence markers to eliminate explana- tion fabrication by encouraging extractive gen- eration, 2) using the fusion-in-decoder archi- tecture to handle long input contexts, and 3) intermediate fine-tuning on re-structured open domain QA datasets to improve few-shot per- formance. FiD-Ex significantly improves over prior work in terms of explanation metrics and task accuracy on five tasks from the ERASER explainability benchmark in both fully super- vised and few-shot settings. 1 Introduction While large pre-trained language models (Devlin et al., 2019; Raffel et al., 2019; Lewis et al., 2020) with hundreds of millions of parameters have made super-human performance possible on various NLP datasets, they lack transparency into their decision making process, which can adversely affect user trust in their predictions. Recent works have pro- posed the use of natural language (NL) rationales (Lei et al., 2016; DeYoung et al., 2020; Latcinnik and Berant, 2020) as a means to either obtain an un- derstanding of the reasoning process of models, or ∗Equal Contribution. 1github.com/facebookresearch/fidex Q: Is Sanskrit the first language of the world ? Answer: False Q: Where does Frodo live ? Choices: Tunnels, Underground, Somewhere nearby Answer: Somewhere nearby The early Jain scholar … of Sanskrit. Sanskrit belongs to the Indo - European family of languages. It is one of the three ancient documented languages that likely arose from a common root language now referred to as ... … Tasha oohed in awe. I said, "Frodo's been visiting you, eh ?" Malaquez said, "Your pet ?" "Hardly. He lives around here somewhere. I suppose he was attracted to the commotion up the hill." ... Figure 1: Example questions, answers and correspond- ing passages from the BoolQ and MultiRC datasets from the ERASER benchmark (DeYoung et al., 2020). Annotated rationales are highlighted. Note that ratio- nales can be multi-sentence and non-contiguous. as a human-readable snippet for users to verify pre- dictions (Lipton, 2018). Figure 1 presents examples of extractive textual rationales for two QA tasks from the ERASER benchmark (DeYoung et al., 2020)2. Recently, Narang et al. (2020) show that …

Content truncated to 500 words