Obtaining Faithful Interpretations from Compositional Neural Networks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5594–5608 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5594 Obtaining Faithful Interpretations from Compositional Neural Networks Sanjay Subramanian∗1 Ben Bogin∗2 Nitish Gupta∗3 Tomer Wolfson1,2 Sameer Singh4 Jonathan Berant1,2 Matt Gardner1 1Allen Institute for AI 2Tel-Aviv University 3University of Pennsylvania 4University of California, Irvine {sanjays,mattg}@allenai.org, {ben.bogin,joberant}@cs.tau.ac.il, nitishg@seas.upenn.edu, tomerwol@mail.tau.ac.il, sameer@uci.edu Abstract Neural module networks (NMNs) are a pop- ular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflect- ing the compositional structure of the prob- lem in the network architecture. However, prior work implicitly assumed that the struc- ture of the network modules, describing the abstract reasoning process, provides a faith- ful explanation of the model’s reasoning; that is, that all modules perform their intended be- haviour. In this work, we propose and con- duct a systematic evaluation of the intermedi- ate outputs of NMNs on NLVR2 and DROP, two datasets which require composing multi- ple reasoning steps. We find that the inter- mediate outputs differ from the expected out- put, illustrating that the network structure does not provide a faithful explanation of model be- haviour. To remedy that, we train the model with auxiliary supervision and propose partic- ular choices for module architecture that yield much better faithfulness, at a minimal cost to accuracy. 1 Introduction Models that can read text and reason about it in a particular context (such as an image, a paragraph, or a table) have been recently gaining increased at- tention, leading to the creation of multiple datasets that require reasoning in both the visual and textual domain (Johnson et al., 2016; Suhr et al., 2017; Talmor and Berant, 2018; Yang et al., 2018a; Suhr et al., 2019; Hudson and Manning, 2019; Dua et al., 2019). Consider the example in Figure 1 from NLVR2: a model must understand the composi- tional sentence in order to then ground dogs in the input, count those that are black and verify that the count of all dogs in the image is equal to the number of black dogs. ∗Equal Contribution “All the dogs are black.” find[dogs] filter[black] 24% 96% Basic-NMN Faithful-NMN equal 100% 100% 76% 100% 13% 100% False (57%) count equal count False (98%) 2 0.9 count count 1.6 1.4 find[dogs] filter[black] Figure 1: An example for a visual reasoning prob- lem where both the Basic and Faithful NMNs produce the correct answer. The Basic NMN, however, fails to give meaningful intermediate outputs for the find and filter modules, whereas our improved Faithful- NMN assigns correct probabilities in all cases. Boxes are green if probabilities are as expected, red otherwise. Both models that assume an intermediate struc- ture (Andreas et al., 2016; Jiang and Bansal, 2019) and models without such structure (Tan and Bansal, 2019; Hu et al., 2019; Min et al., 2019) have been proposed for these reasoning problems. While good performance can be obtained without a struc- tured representation, an advantage of …

Content truncated to 500 words