Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2810–2829 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2810 Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs Ana Marasovi´c†♦ Chandra Bhagavatula† Jae Sung Park♦† Ronan Le Bras† Noah A. Smith♦† Yejin Choi♦† †Allen Institute for Artificial Intelligence ♦Paul G. Allen School of Computer Science & Engineering, University of Washington Seattle, WA, USA {anam,chandrab,jamesp,ronanlb}@allenai.org {jspark96,nasmith,yejin}@cs.washington.edu Abstract Natural language rationales could provide in- tuitive, higher-level explanations that are eas- ily understandable by humans, complementing the more broadly studied lower-level explana- tions based on gradients or attention weights. We present the first study focused on generat- ing natural language rationales across several complex visual reasoning tasks: visual com- monsense reasoning, visual-textual entailment, and visual question answering. The key chal- lenge of accurate rationalization is comprehen- sive image understanding at all levels: not just their explicit content at the pixel level, but their contextual contents at the semantic and pragmatic levels. We present RATIONALEVT TRANSFORMER, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recog- nition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that free-text rationalization is a promis- ing research direction to complement model interpretability for complex visual-textual rea- soning tasks. In addition, we find that integra- tion of richer semantic and pragmatic visual features improves visual fidelity of rationales. 1 Introduction Explanatory models based on natural language ra- tionales could provide intuitive, higher-level expla- nations that are easily understandable by humans (Miller, 2019). In Figure 1, for example, the natu- ral language rationale given in free-text provides a much more informative and conceptually relevant explanation to the given QA problem compared to the non-linguistic explanations that are often pro- vided as localized visual highlights on the image. The latter, while pertinent to what the vision com- ponent of the model was attending to, cannot pro- vide the full scope of rationales for such complex reasoning tasks as illustrated in Figure 1. Indeed, explanations for higher-level conceptual reasoning can be best conveyed through natural language, as has been studied in recent literature on (visual) NLI (Do et al., 2020; Camburu et al., 2018), (visual) QA (Wu and Mooney, 2019; Rajani et al., 2019), playing arcade games (Ehsan et al., 2019), fact checking (Atanasova et al., 2020), image classifica- tion (Hendricks et al., 2018), motivation prediction (Vondrick et al., 2016), and self-driving cars (Kim et al., 2018). In this paper, we present the first focused study on generating natural language rationales across several complex visual reasoning tasks: visual com- monsense reasoning, visual-textual entailment, and visual question answering. Our study aims to com- plement the more broadly studied lower-level ex- planations such as attention weights and gradients in deep neural networks (Simonyan et al., 2014; Zhang et al., 2017; Montavon et al., 2018, among others). Because free-text rationalization is a chal- lenging research question, we assume the gold an- swer for a given instance …

Content truncated to 500 words