ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activities

Findings of the Association for Computational Linguistics: EACL 2023, pages 2192–2207 May 2-6, 2023 ©2023 Association for Computational Linguistics ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activities Terry Yue Zhuo1 and Yaqing Liao2 and Yuecheng Lei2 Lizhen Qu1* and Gerard de Melo3 Xiaojun Chang4 and Yazhou Ren2 and Zenglin Xu5,6* 1Monash University 2University of Electronic Science and Technology of China 3HPI/Univerisity of Potsdam 4University of Technology Sydney 5Harbin Institute of Technology, Shenzhen 6Peng Cheng Lab Abstract We introduce ViLPAct, a novel vision- language benchmark for human activity plan- ning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their ini- tial activities and intents in text. The dataset consists of 2.9k videos from Charades ex- tended with intents via crowdsourcing, a multi- choice question test set, and four strong base- lines. One of the baselines implements a neu- rosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods. According to our extensive experiments, the key challenges are compositional generalization and effective use of information from both modalities1. 1 Introduction "He wants to keep his food fresh." Intent The old man is now standing in the kitchen. Holding some food Putting some food somewhere Holding some clothes Opening a refrigerator Holding a knife Cook some food Opening a refrigerator Putting some food somewhere Holding some food What should come next? Observation Figure 1: In daily life scenarios, an agent should be aware of future actions that will likely be taken by the user based on what it has observed. In this example, inputs of intent and observation are colored in green, while potential future action sequences are highlighted in orange. The first two sequences contain actions which do not align with the human intent. Thus, the agent needs to automatically detect which future actions are plausible by understanding the user’s intent. One of the ultimate goals of Artificial Intelli- gence is to build intelligent agents capable of accu- rately understanding humans’ actions and intents, *Corresponding authors: lizhen.qu@monash.edu, xuzenglin@hit.edu.cn 1Our benchmark is available at https://github. com/terryyz/ViLPAct so that they can better serve us (Kong and Fu, 2018). Newly emerging applications in robotics and multi-modal planning, such as Amazon Astro, have demonstrated a strong need to understand hu- man behavior in multimodal environments. On the one hand, such an agent, e.g. an elderly care ser- vice bot, needs to understand human activities and anticipate human behaviors based on users’ intents. Here the intents may be estimated based on previ- ous activities or articulated verbally by users. The anticipated behaviors may be used for risk assess- ment (e.g. falling of elderly people) and to facilitate collaboration with humans. On the other hand, re- cent advances in robotics show that it is possible to let robots learn new tasks directly from observed human behavior without robot demonstrations (Yu et al., 2018; Sharma et al., 2019). However, …

Content truncated to 500 words