Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 861–877, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 861 Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! Jack Hessel Allen Institute for AI jackh@allenai.org Lillian Lee Cornell University llee@cs.cornell.edu Abstract Modeling expressive cross-modal interactions seems crucial in multimodal tasks, such as visual question answering. However, some- times high-performing black-box algorithms turn out to be mostly exploiting unimodal sig- nals in the data. We propose a new diagnos- tic tool, empirical multimodally-additive func- tion projection (EMAP), for isolating whether or not cross-modal interactions improve per- formance for a given model on a given task. This function projection modifies model pre- dictions so that cross-modal interactions are eliminated, isolating the additive, unimodal structure. For seven image+text classification tasks (on each of which we set new state-of- the-art benchmarks), we find that, in many cases, removing cross-modal interactions re- sults in little to no performance degradation. Surprisingly, this holds even when expressive models, with capacity to consider interactions, otherwise outperform less expressive models; thus, performance improvements, even when present, often cannot be attributed to con- sideration of cross-modal feature interactions. We hence recommend that researchers in mul- timodal machine learning report the perfor- mance not only of unimodal baselines, but also the EMAP of their best-performing model. 1 Introduction Given the presumed importance of reasoning across modalities in multimodal machine learn- ing tasks, we should evaluate a model’s ability to leverage cross-modal interactions. But such evalu- ation is not straightforward; for example, an early Visual Question-Answering (VQA) challenge was later “broken” by a high-performing method that ignored the image entirely (Jabri et al., 2016). One response is to create multimodal-reasoning datasets that are specifically and cleverly balanced to resist language-only or visual-only models; ex- amples are VQA 2.0 (Goyal et al., 2017), NLVR2 Multimodally-additive models Linear model An image+text ensemble Kernel SVM LXMERT Neural Net Empirical Multimodally Additive Projection Figure 1: We introduce EMAP, a diagnostic for clas- sifiers that take in textual and visual inputs. Given a (black-box) trained model, EMAP computes the pre- dictions of an image/text ensemble that best approx- imates the full model predictions via empirical func- tion projection. Although the projected predictions lose visual-textual interaction signals exploited by the orig- inal model, they often perform suprisingly well. (Suhr et al., 2019), and GQA (Hudson and Man- ning, 2019). However, a balancing approach not always desirable. For example, if image+text data is collected from an online social network (such as for popularity prediction or sentiment analy- sis), post-hoc rebalancing may obscure trends in the original data-generating processs. So, what al- ternative diagnostic tools are available for better understanding what models learn? The main tool utilized by prior work is model comparison. In addition to comparing against text-only and image-only baselines, often, two multimodal models with differing representational capacity (e.g., a cross-modal attentional neural network vs. a linear model) are trained and their performance compared. The argument commonly made …

Content truncated to 500 words