Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 932 - 946 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues Qingxiu Dong1 ∗, Ziwei Qin1 ∗, Heming Xia1, Tian Feng1, Shoujie Tong1, Haoran Meng1, Lin Xu1, Zhongyu Wei2, Weidong Zhan1, Baobao Chang1, Sujian Li1, Tianyu Liu3, Zuifang Sui1 1 Key Laboratory of Computational Linguistics, Peking University, MOE, China 2 School of Data Science, Fudan University 3 Tencent Cloud Xiaowei {dqx,qinziwei}@stu.pku.edu.cn Abstract It is a common practice for recent works in vi- sion language cross-modal reasoning to adopt a binary or multi-choice classification formu- lation taking as input a set of source image(s) and textual query. In this work, we take a sober look at such an “unconditional” formulation in the sense that no prior knowledge is specified with respect to the source image(s). Inspired by the designs of both visual commonsense rea- soning and natural language inference tasks, we propose a new task termed “Premise-based Multi-modal Reasoning” (PMR) where a tex- tual premise is the background presumption on each source image. The PMR dataset contains 15,360 manually annotated samples which are created by a multi-phase crowd-sourcing pro- cess. With selected high-quality movie screen- shots and human-curated premise templates from 6 pre-defined categories, we ask crowd- source workers to write one true hypothesis and three distractors (4 choices) given the premise and image through a cross-check procedure. Besides, we generate adversarial samples to alleviate the annotation artifacts and double the size of PMR. We benchmark various state- of-the-art (pretrained) multi-modal inference models on PMR and conduct comprehensive experimental analyses to showcase the utility of our dataset. 1 Introduction Cross-modal reasoning between image and text has been recognized as a fundamental and long- standing task in both academia and industry, which has recently attracted intensive attention from both natural language processing and computer vision communities (Su et al., 2019; Yu et al., 2020; Chen et al., 2020). Researchers try to teach machines to perceive, understand and reason with both visual and textual clues, which mimics human cognitive process (Lake et al., 2016). *Equal contribution. The canonical form for the cross-modal reason- ing tasks usually take the source image(s) as the input and request the inference model to perform a multi-choice classification according to the speci- fied textual query, including visual question answer- ing (Antol et al., 2015; Goyal et al., 2017), visual commonsense reasoning (Zellers et al., 2018), vi- sual entailment (Xie et al., 2019) and image-text grounding (Suhr et al., 2017, 2019), etc. We ar- gue the default cross-modal setting is “static” or “unconditional” in the sense that no prior presump- tion or belief is attached to the input images (Ren et al., 2021). The lack of prior presumption would be insignificant while answering factoid questions according to the images, e.g. “how many dogs are lying on the grass?” (Suhr et al., 2019) or “is the bowl to the right of the …

Content truncated to 500 words