CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Findings of the Association for Computational Linguistics: ACL 2022, pages 2602 - 2627 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions Tayfun Ates1,∗ M. Samil Atesoglu1,∗ Cagatay Yigit1,∗ Ilker Kesen2,3 Mert Kobas4 Erkut Erdem1,2 Aykut Erdem2,3 Tilbe Goksun4 Deniz Yuret2,3 1 Hacettepe University, Computer Engineering Department 2 Koç University, KUIS AI Center 3 Koç University, Computer Engineering Department 4 Koç University, Psychology Department Abstract Humans are able to perceive, understand and reason about causal events. Developing mod- els with similar physical and causal under- standing capabilities is a long-standing goal of artificial intelligence. As a step towards this direction, we introduce CRAFT1, a new video question answering dataset that requires causal reasoning about physical forces and object in- teractions. It contains 58K video and ques- tion pairs that are generated from 10K videos from 20 different virtual environments, con- taining various objects in motion that interact with each other and the scene. Two question categories in CRAFT include previously stud- ied descriptive and counterfactual questions. Additionally, inspired by the Force Dynamics Theory in cognitive linguistics, we introduce a new causal question category that involves understanding the causal interactions between objects through notions like cause, enable, and prevent. Our results show that even though the questions in CRAFT are easy for humans, the tested baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark. 1 Introduction Causal reasoning is a key cognitive capability that involves making predictions about physical objects and their interactions. Cognitive scientists have mainly studied causal reasoning as simple causes or chains of events (Michotte, 1963; Baillargeon, 1994; Saxe et al., 2005), rather than processing of complex causal scenes, see (Göksun et al., 2013; George et al., 2019). Referring to the interactions of multiple forces, the Force Dynamics Theory em- phasizes the processing and reasoning of complex scenes, and how causal language defines the pat- terns of forces in causal events (Wolff, 2007). ∗indicates equal contributions. 1Data and code available on our project website at https://sites.google.com/view/craft-benchmark In the past decade, though artificial learning sys- tems have shown astonishing progress in natural language and image understanding, there are some tasks in which these systems are still significantly below human performance. One such challenging research area includes reasoning about physical ac- tions of objects in complex causal scenes. In this paper, we explore how language and vision inter- act with each other in making plausible projections about causal reasoning, and analyze how well the existing neural models understand and reason about physical and causal relationships between dynamic objects in a scene through images and text. We propose a new video question answering dataset, named CRAFT (Causal Reasoning About Forces and inTeractions), which is designed to be complex for artificial models and simple for hu- mans. Our dataset contains synthetically generated videos of 2D scenes with accompanying questions. Its most prominent features are that it contains video clips with complex physical interactions be- tween …

Content truncated to 500 words