Situated and Interactive Multimodal Conversations

Proceedings of the 28th International Conference on Computational Linguistics, pages 1103–1121 Barcelona, Spain (Online), December 8-13, 2020 1103 Situated and Interactive Multimodal Conversations Seungwhan Moon∗, Satwik Kottur∗, Paul A. Crook†, Ankita De†, Shivani Poddar† Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami Eunjoon Cho, Rajen Subba, Alborz Geramifard Facebook � simmc@fb.com Abstract Next generation virtual assistants are envisioned to handle multimodal inputs (e.g., vision, mem- ories of previous interactions, and the user’s utterances), and perform multimodal actions (e.g., displaying a route while generating the system’s utterance). We introduce Situated Interactive MultiModal Conversations (SIMMC) as a new direction aimed at training agents that take mul- timodal actions grounded in a co-evolving multimodal input context in addition to the dialog history. We provide two SIMMC datasets totalling ∼13K human-human dialogs (∼169K utter- ances) collected using a multimodal Wizard-of-Oz (WoZ) setup, on two shopping domains: (a) furniture – grounded in a shared virtual environment; and (b) fashion – grounded in an evolv- ing set of images. Datasets include multimodal context of the items appearing in each scene, and contextual NLU, NLG and coreference annotations using a novel and unified framework of SIMMC conversational acts for both user and assistant utterances. Finally, we present several tasks within SIMMC as objective evaluation protocols, such as struc- tural API prediction, response generation, and dialog state tracking. We benchmark a collection of existing models on these SIMMC tasks as strong baselines, and demonstrate rich multimodal conversational interactions. Our data, annotations, and models are publicly available.1 1 Introduction As virtual digital assistants become increasingly ubiquitous, they are expected to be embedded in the day-to-day life of users the same way a human assistant would. We thus envision that the next generation of virtual assistants will be able to process multimodal inputs and provide multimodal outputs beyond the traditional NLP stack. To this end, we present Situated Interactive MultiModal Conversations (SIMMC) tasks and datasets as a starting point in this new research direction. Specifically, SIMMC focuses on task-oriented dialogs that encompass a situated multimodal context, where situated im- plies that the user and assistant are continually co-observing the same context, and that context can be updated on each turn. We provide two new SIMMC datasets in the domain of interactive shopping, collected using the SIMMC Platform (Crook et al., 2019): (1) Furniture and (2) Fashion. Moreover, we provide fine-grained annotations to allow for both end-to-end and component-level modelling. The annotation includes natural language understanding (NLU), multimodal-coreference, multimodal state tracking, assistant actions, natural language generation (NLG), and item appearance logs. Fig. 1 illustrates an exemplary dialog from our SIMMC-Furniture dataset, where a user is interacting with an assistant with the goal of browsing for furniture. In our setting, the assistant can update the co- observed environment to create a new multimodal context based on the preceding dialog with the user, e.g., visually presenting recommended chairs in a virtual reality (VR) environment, or responding to the request “I like the brown one. Show me the back of it.” by executing the …

Content truncated to 500 words