SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6273–6291 July 9-14, 2023 ©2023 Association for Computational Linguistics SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams Te-Lin Wu1∗, Satwik Kottur2, Andrea Madotto2, Mahmoud Azab2 Pedro Rodriguez3, Babak Damavandi2, Nanyun Peng1, Seungwhan Moon2 1University of California, Los Angeles, 2Meta Reality Labs, 3Meta AI {telinwu,violetpeng}@cs.ucla.edu, {skottur,andreamad8,azab,par,babakd,shanemoon}@meta.com Abstract Building an AI assistant that can seamlessly converse and instruct humans, in a user-centric situated scenario, requires several essential abil- ities: (1) spatial and temporal understanding of the situated and real-time user scenes, (2) ca- pability of grounding the actively perceived vi- suals of users to conversation contexts, and (3) conversational reasoning over past utterances to perform just-in-time assistance. However, we currently lack a large-scale benchmark that captures user↔assistant interactions with all of the aforementioned features. To this end, we propose SIMMC-VR, an extension of the SIMMC 2.0 dataset to a video-grounded task- oriented dialog dataset that captures real-world AI-assisted user scenarios in VR. We propose a novel data collection paradigm that involves (1) generating object-centric multimodal dia- log flows with egocentric visual streams and visually-grounded templates, and (2) manually paraphrasing the simulated dialogs for natural- ness and diversity while preserving multimodal dependencies. To measure meaningful progress in the field, we propose four tasks to address the new challenges in SIMMC-VR, which require complex spatial-temporal dialog reasoning in active egocentric scenes. We benchmark the proposed tasks with strong multimodal models, and highlight the key capabilities that current models lack for future research directions. 1 Introduction With the growing popularity of smart glasses, stud- ies on visually grounded conversational agents have gained significant interest. For instance, SIMMC 2.0 (Kottur et al., 2021) introduces an image- grounded, task-oriented dialog (TOD) dataset where an assistant agent co-observes the user’s ego- centric viewpoint to aid with user requests. Many follow-up works (Huang et al., 2021a; Lee et al., 2022; Chiyah-Garcia et al., 2022) focus on chal- lenges around dialog-image grounding, such as ∗Work done during an internship at Meta Reality Labs. I recommend the black coat in front of you on the wall. INFORM:GET:COAT Dialog Acts Multimodal Coref. I would want a coat that goes well with the blue dress I liked. USER Assistant INFORM:COMPL:COAT Do you have something like it but in light grey? Yes, pivot to your left and move forward … You can get closer to the light grey coat. REQUEST:REFINE:COAT NAVIGATIONAL INSTR. Frame 12. Frame 178. Frame 339. Frame 364. Frame 422. Frame 451. I like the black coat better than this one, is it my size? I think so, the coat is a large and comes with other sizes, too. INFORM:PREFER:COAT INFORM:GET:COAT.size Figure 1: SIMMC-VR is a Situated Interactive Multi- Modal Conversation dataset that features task-oriented user↔assistant dialogs streamed immersively in a virtual- reality (VR) environment. The dataset is created on pro- grammed realistic shopping scenarios and actively-rendered photorealistic user visual observations, which brings new chal- lenges for complex spatial-temporal reasoning on …

Content truncated to 500 words