A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10757–10770 July 9-14, 2023 ©2023 Association for Computational Linguistics A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues Yunxin Li1, Baotian Hu1∗, Xinyu Chen1, Yuxin Ding1, Lin Ma2, Min Zhang1 1Harbin Institute of Technology, Shenzhen, China, 2Meituan, Beijing {hubaotian, yxding, zhangmin2021}@hit.edu.cn {liyunxin987, chenxinyuhitsz}@163.com, forest.linma@gmail.com Abstract Conditional inference on joint textual and visual clues is a multi-modal reasoning task that textual clues provide prior permutation or external knowledge, which are complementary with visual content and pivotal to deducing the correct option. Previous methods utilizing pretrained vision-language models (VLMs) have achieved impressive performances, yet they show a lack of multimodal context reasoning capability, especially for text-modal information. To address this issue, we propose a Multi-modal Context Reasoning approach, named ModCR. Compared to VLMs performing reasoning via cross modal semantic alignment, it regards the given textual abstract semantic and objective image information as the pre-context information and embeds them into the language model to perform context reasoning. Different from recent vision-aided language models used in natural language processing, ModCR incorporates the multi-view semantic alignment information between language and vision by introducing the learnable alignment prefix between image and text in the pretrained language model. This makes the language model well-suitable for such multi-modal reasoning scenario on joint textual and visual clues. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved perfor- mance (exact gain by 4.8% on PMR test set) compared to previous strong baselines. Code Link: https://github.com/YunxinLi/ Multimodal-Context-Reasoning. 1 Introduction Cross modal reasoning is a hot research topic both in natural language processing and computer vi- sion communities. Most cross modal reasoning tasks, such as Visual Question Answering (Antol et al., 2015; Wu et al., 2017; Shah et al., 2019; ∗ Corresponding author. Figure 1: A case from the PMR (Dong et al., 2022) data set, where the correct option is answer B. The blue- color words represent the pivotal textual clue to infer the correctness of answers A and B. Yusuf et al., 2022), Visual Dialog (Zhang et al., 2022; Chen et al., 2022), Visual Entailment, (Xie et al., 2019; Do et al., 2020) and Visual Common- sense Reasoning (Zellers et al., 2019a; Ye and Ko- vashka, 2021; Li et al., 2022a), concentrate on the visual reasoning scenario that relies primarily on image information. The given text (or question) is highly attached to the image and lacks prior permu- tation, e.g., the common question “Why is person 4 pointing to person 1" shown in VCR (Zellers et al., 2019a) data set. For another practical cross modal reasoning scenario (Dong et al., 2022), the textual modality often provides prior permutation or complementary information with the source im- age, such as the commonsense knowledge, and the personalities, feelings, or relationships of persons, as the premise shown in Figure 1. In this paper, we focus on such conditional inference on joint textual and visual clues, where …

Content truncated to 500 words