CoSIm: Commonsense Reasoning for Counterfactual Scene Imagination

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 911 - 923 July 10-15, 2022 ©2022 Association for Computational Linguistics COSIM: Commonsense Reasoning for Counterfactual Scene Imagination Hyounghun Kim* Abhay Zala* Mohit Bansal UNC Chapel Hill {hyounghk, aszala, mbansal}@cs.unc.edu Abstract As humans, we can modify our assumptions about a scene by imagining alternative objects or concepts in our minds. For example, we can easily anticipate the implications of the sun be- ing overcast by rain clouds (e.g., the street will get wet) and accordingly prepare for that. In this paper, we introduce a new dataset called Commonsense Reasoning for Counterfactual Scene Imagination (COSIM) which is designed to evaluate the ability of AI systems to rea- son about scene change imagination. To be specific, in this multimodal task/dataset, mod- els are given an image and an initial question- response pair about the image. Next, a counter- factual imagined scene change (in textual form) is applied, and the model has to predict the new response to the initial question based on this scene change. We collect 3.5K high-quality and challenging data instances, with each in- stance consisting of an image, a commonsense question with a response, a description of a counterfactual change, a new response to the question, and three distractor responses. Our dataset contains various complex scene change types (such as object addition/removal/state change, event description, environment change, etc.) that require models to imagine many dif- ferent scenarios and reason about the changed scenes. We present a baseline model based on a vision-language Transformer (i.e., LXMERT) and ablation studies. Through human evalua- tion, we demonstrate a large human-model per- formance gap, suggesting room for promising future work on this challenging, counterfactual multimodal task.1 1 Introduction Anticipating what would happen when there is a condition change in a situation is an important abil- ity as it allows preparation for the implications of *Equal contribution. 1Our code and dataset are publicly available at: https: //github.com/hyounghk/CoSIm. the change. For example, when swimming in the sea on a clear day, you might feel safe. However, if someone told you a storm warning has been issued and dark clouds are coming in soon, you would know that it is no longer safe to swim and return to land. It will be also very useful to have AI systems that could reason about the implications of such scenario changes and provide appropriate guidance/warnings; however, current AI systems will have a hard time performing such counterfac- tual commonsense reasoning. Many efforts have been made to teach machines how to reason about images (Antol et al., 2015; Zhu et al., 2016; Johnson et al., 2017; Hudson and Manning, 2019) and videos (Tapaswi et al., 2016; Jang et al., 2017; Zhu et al., 2017; Lei et al., 2018). This area has been built upon further by efforts to teach machines to use commonsense knowledge when analyzing visual scenes (Pirsiavash et al., 2014; Wagner et al., 2018; Zellers et al., …

Content truncated to 500 words