SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4903–4912 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4903 SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations Satwik Kottur⇤, Seungwhan Moon⇤, Alborz Geramifard, Babak Damavandi Facebook Reality Labs & Facebook AI {skottur,shanemoon,alborzg,babakd}@fb.com Abstract Next generation task-oriented dialog systems need to understand conversational contexts with their perceived surroundings, to effec- tively help users in the real-world multi- modal environment. Existing task-oriented di- alog datasets aimed towards virtual assis- tance fall short and do not situate the di- alog in the user’s multimodal context. To overcome, we present a new dataset for Sit- uated and Interactive Multimodal Conversa- tions, SIMMC 2.0, which includes 11K task- oriented user$assistant dialogs (117K utter- ances) in the shopping domain, grounded in immersive and photo-realistic scenes. The dialogs are collected using a two-phase pipeline: (1) A novel multimodal dialog sim- ulator generates simulated dialog flows, with an emphasis on diversity and richness of inter- actions, (2) Manual paraphrasing of the gener- ated utterances to collect diverse referring ex- pressions. We provide an in-depth analysis of the collected dataset, and describe in detail the four main benchmark tasks we propose. Our baseline model, powered by the state-of-the- art language model, shows promising results, and highlights new challenges and directions for the community to study 1. 1 Introduction The Situated and Interactive Multimodal Conversa- tional AI (SIMMC) challenge (Moon et al., 2020), held as part of DSTC9 (Gunasekara et al., 2020), aimed to lay the foundations for the real-world as- sistant agents that can handle multimodal inputs, and perform multimodal actions. Specifically, it provided the SIMMC datasets as new benchmarks for studying task-oriented dialogs that encompass a situated multimodal user context in the form of a co-observed image or virtual reality (VR) envi- ronment. Since the introduction of the dataset, a ⇤Joint first authors 1Code & data are made publicly available: https:// github.com/facebookresearch/simmc2 Figure 1: Illustration of a Situated Interactive Mul- timodal Conversation (SIMMC), which presents a task-oriented user$assistant dialog grounded in a co- observed multimodal context. The newly collected SIMMC 2.0 dataset includes complex and photorealis- tic multimodal contexts, which poses more challenges for the Multimodal Coreference Resolution task (MM- Coref) and the Multimodal Dialog State Tracking task. number of follow-up work (Kung et al., 2021; Kim et al., 2021; Jeong et al., 2021; Huang et al., 2021; Senese et al., 2021) have established a new set of state-of-the-art baselines for the multimodal task- oriented dialog systems on SIMMC. Though SIMMC serves as a step towards build- ing multimodal virtual agents, the dataset falls short (understandably so) in the complexity of the con- sidered multimodal contexts. In particular, the co- observed image or VR environment is simplistic and far from realistic user situations. To bridge this gap, we take inspiration from the first SIMMC 4904 challenge (Moon et al., 2020) and propose a new multimodal dialog dataset (SIMMC 2.0) for the com- munity to tackle and continue the effort towards building a …

Content truncated to 500 words