Multi-modal Action Chain Abductive Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4617–4628 July 9-14, 2023 ©2023 Association for Computational Linguistics Multi-modal Action Chain Abductive Reasoning Mengze Li1, Tianbao Wang1, Jiahe Xu1, Kairong Han1, Shengyu Zhang1, Zhou Zhao1∗, Jiaxu Miao1∗, Wenqiao Zhang1∗, Shiliang Pu4, Fei Wu2,3 1Zhejiang University 2Shanghai Institute for Advanced Study of Zhejiang University 3Shanghai AI Laboratory 4Hikvision Research Institute Abstract Abductive Reasoning, has long been considered to be at the core ability of humans, which en- ables us to infer the most plausible explanation of incomplete known phenomena in daily life. However, such critical reasoning capability is rarely investigated for contemporary AI sys- tems under such limited observations. To facil- itate this research community, this paper sheds new light on Abductive Reasoning by study- ing a new vision-language task, Multi-modal Action chain abductive Reasoning (MAR), to- gether with a large-scale Abductive Reason- ing dataset: Given an incomplete set of lan- guage described events, MAR aims to imagine the most plausible event by spatio-temporal grounding in past video and then infer the hy- pothesis of subsequent action chain that can best explain the language premise. To solve this task, we propose a strong baseline model that realizes MAR from two perspectives: (i) we first introduce the transformer, which learns to encode the observation to imagine the plau- sible event with explicitly interpretable event grounding in the video based on the common- sense knowledge recognition ability. (ii) To complete the assumption of a follow-up action chain, we design a novel symbolic module that can complete strict derivation of the progres- sive action chain layer by layer. We conducted extensive experiments on the proposed dataset, and the experimental study shows that the pro- posed model significantly outperforms existing video-language models in terms of effective- ness on our newly created MAR dataset. Our dataset is available 1. First author. mengzeli@zju.edu.cn ∗Corresponding author. zhaozhou@zju.edu.cn jiaxu.miao@yahoo wenqiaozhang@zju.edu.cn 1https://github.com/BirdFly16/TO-MAR Observation: Action Chain: The man in a T-shirt choked on food is vomiting into the toilet. chock put down stand up run out run in bend over Event Grounding: Abductive Reasoning Timeline Explanation: (devour the food) t0 t1 t2 Multi-modal Action Chain Abductive Reasoning Figure 1: A diagram of the MAR task. 1 Introduction Abductive Reasoning typically begins with an in- complete observation or several observations and then proceeds to the likeliest possible explanation for the set (Du et al., 2021; Peirce, 1974). Given an event observation (O), humans can find some related information in the recollection and easily trace the complete process with strong reasoning ability as a hypothesis (H) to explain the observa- tion. For instance, when we observe the O: "The man in a T-shirt chocked on food is vomiting into the toilet" and remember that he used to devour food in the kitchen, we could infer the complete event chain about that man as hypothesis H: the man devoured the food in the kitchen → he was choking → he put down the food → he …

Content truncated to 500 words