CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9856 - 9870 December 7-11, 2022 ©2022 Association for Computational Linguistics CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering Maitreya Patel and Tejas Gokhale and Chitta Baral and Yezhou Yang Arizona State University {maitreya.patel, tgokhale, chitta, yz.yang}@asu.edu Abstract Videos often capture objects, their visible prop- erties, their motion, and the interactions be- tween different objects. Objects also have phys- ical properties such as mass, which the imaging pipeline is unable to directly capture. However, these properties can be estimated by utilizing cues from relative object motion and the dy- namics introduced by collisions. In this pa- per, we introduce CRIPP-VQA1, a new video question answering dataset for reasoning about the implicit physical properties of objects in a scene. CRIPP-VQA contains videos of objects in motion, annotated with questions that in- volve counterfactual reasoning about the effect of actions, questions about planning in order to reach a goal, and descriptive questions about visible properties of objects. The CRIPP-VQA test set enables evaluation under several out- of-distribution settings – videos with objects with masses, coefficients of friction, and initial velocities that are not observed in the training distribution. Our experiments reveal a surpris- ing and significant performance gap in terms of answering questions about implicit properties (the focus of this paper) and explicit properties of objects (the focus of prior work). 1 Introduction Visual grounding seeks to link images or videos with natural language. Towards this goal, many tasks such as referring expressions (Yu et al., 2016), captioning (Vinyals et al., 2015; Xu et al., 2016), text-based retrieval (Vo et al., 2019; Rohrbach et al., 2015), and visual question answering (Antol et al., 2015; Jang et al., 2017) have been studied for both images and videos. Videos often contain objects which can be identified in terms of their visible properties such as their shapes, sizes, colors, tex- tures, and categories. These visible properties can be estimated by using computer vision algorithms for object recognition, detection, color recognition, 1https://maitreyapatel.com/CRIPP-VQA/ Figure 1: The CRIPP-VQA dataset contains questions about the future effect of actions (such as removing, adding, or replacing objects) as well as planning-based questions. Frames from an example video are shown above with the red highlighted area depicting the objects on which actions (remove, replace, add) are performed. shape estimation, etc. However, objects also have physical properties such as mass and coefficient of friction, which are not captured by cameras. For instance, given a video of a stone rolling down a hill, cameras can capture the color of the stone and its trajectory – but how heavy is the stone? It is therefore difficult to reason about such implicit physical properties, by simply watching videos. Collisions between objects, however, do offer 9856 visual cues about mass and friction. When objects collide, their resulting velocities and directions of motion depend upon their physical properties, and are governed by fundamental laws of physics such as conservation of momentum and …

Content truncated to 500 words