Debiasing Event Understanding for Visual Commonsense Tasks

Findings of the Association for Computational Linguistics: ACL 2022, pages 782 - 787 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Debiasing Event Understanding for Visual Commonsense Tasks Minji Seo1∗, YeonJoon Jung2∗, Seungtaek Choi2, Seung-won Hwang1†, Bei Liu3 1Seoul National University, 2Yonsei University, 3Microsoft Research Asia {minjiseo, seungwonh}@snu.ac.kr, {theaitetus, hist0613}@yonsei.ac.kr, Bei.Liu@microsoft.com Abstract We study event understanding as a critical step towards visual commonsense tasks. Meanwhile, we argue that current object-based event under- standing is purely likelihood-based, leading to incorrect event prediction, due to biased corre- lation between events and objects. We propose to mitigate such biases with do-calculus, pro- posed in causality research, but overcoming its limited robustness, by an optimized aggre- gation with association-based prediction. We show the effectiveness of our approach, intrin- sically by comparing our generated events with ground-truth event annotation, and extrinsically by downstream commonsense tasks. 1 Introduction Recently, commonsense reasoning tasks on visio- linguistic input have been actively studied in both vision and language communities, with the goal of commonsense reasoning. For example, in Visual COMET (Park et al., 2020), given an image X and event e as input, we are tasked to predict intent (or, events before/after), known as intent predic- tion. Similarly, Visual Commonsense Reasoning (VCR; Zellers et al. 2019), given image and ques- tion Q, requires to provide an answer A or provide a rationale for A, known as justification task. For such tasks, understanding event e plays a crucial role, either required as input or expected as output–for example, 41% of rationales from VCR are related to e. However, assuming the availabil- ity of e for new images at test time is considered impractical, for which Park et al. (2020) consider two baselines: 1) setting input e as NULL, and 2) generating e by training generator GEN. However, according to Park et al. (2020), empirical results reported GEN performs even worse than NULL, or, shows a negative transfer, which is counter- intuitive. ∗Equal contribution †Corresponding author (GOLD) “3 kneels next to the bed 1 is lying in” / (confounder) chair (B-GEN) “sit up from the couch” (D-GEN) “lying in the bed” 𝑷(𝒔𝒊𝒔���|𝒅𝒐 𝒐𝒃𝒊��� , 𝒊𝒎𝒈) 𝑷(𝒔𝒊𝒔���|𝒐𝒃𝒊���, 𝒊𝒎𝒈) chair bed Figure 1: An actual example of event generation. By removing the chair (confounder), the biased gen- eration (B-GEN) can be corrected to the true event (D-GEN). Our goal is robustifying event understanding, or, event generator GEN with a higher transferability to the downstream tasks of visual commonsense reasoning. Our hypothesis is that object-event bias in the dataset hinders transferability: In Figure 1, while the image and human-annotated e, denoted as GOLD, are not related to an event “sitting”, existing GEN, denoted as B-GEN for Biased GENeration, generates an event “sitting”, even when no one is sitting on the chair. We argue that the generation is biased by the frequent association of chair-sitting in the dataset. In contrast, we propose D-GEN for Debiased GENeration. Contrary to association-based estima- tion, which fails to distinguish spurious correlation P(sitting|chair), do-calculus collectively considers other observations, such as …

Content truncated to 500 words