Multimodal Dialogue State Tracking

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3394 - 3415 July 10-15, 2022 ©2022 Association for Computational Linguistics Multimodal Dialogue State Tracking Hung Le†‡§, Nancy F. Chen§, Steven C.H. Hoi†‡ †Salesforce Research Asia ‡Singapore Management University §Agency for Science, Technology and Research (A*STAR) {hungle,shoi}@salesforce.com, nfychen@i2r.a-star.edu.sg Abstract Designed for tracking user goals in dialogues, a dialogue state tracker is an essential compo- nent in a dialogue system. However, the re- search of dialogue state tracking has largely been limited to unimodality, in which slots and slot values are limited by knowledge domains (e.g. restaurant domain with slots of restau- rant name and price range) and are defined by specific database schema. In this paper, we pro- pose to extend the definition of dialogue state tracking to multimodality. Specifically, we in- troduce a novel dialogue state tracking task to track the information of visual objects that are mentioned in video-grounded dialogues. Each new dialogue utterance may introduce a new video segment, new visual objects, or new object attributes and a state tracker is re- quired to update these information slots accord- ingly. We created a new synthetic benchmark and designed a novel baseline, Video-Dialogue Transformer Network (VDTN), for this task. VDTN combines both object-level features and segment-level features and learns contextual dependencies between videos and dialogues to generate multimodal dialogue states. We op- timized VDTN for a state generation task as well as a self-supervised video understanding task which recovers video segment or object representations. Finally, we trained VDTN to use the decoded states in a response prediction task. Together with comprehensive ablation and qualitative analysis, we discovered inter- esting insights towards building more capable multimodal dialogue systems. 1 Introduction The main goal of dialogue research is to develop intelligent agents that can assist humans through conversations. For example, a dialogue agent can be tasked to help users to find a restaurant based on their preferences of price ranges and food choices. A crucial part of a dialogue sys- tem is Dialogue State Tracking (DST), which is responsible for tracking and updating user goals in the form of dialogue states, including a set of (slot, value) pairs such as (price, “moder- ate”) and (food, “japanese”). Numerous machine learning approaches have been proposed to tackle DST, including fixed-vocabulary models (Ramadan et al., 2018; Lee et al., 2019) and open-vocabulary models (Lei et al., 2018b; Wu et al., 2019; Le et al., 2020c), for either single-domain (Wen et al., 2017) or multi-domain dialogues (Eric et al., 2017; Budzianowski et al., 2018). However, the research of DST has largely lim- ited the scope of dialogue agents to unimodality. In this setting, the slots and slot values are defined by the knowledge domains (e.g. restaurant domain) and database schema (e.g. data tables for restaurant entities). The ultimate goal of dialogue research towards building artificial intelligent assistants ne- cessitates DST going beyond unimodal systems. In this paper, we propose Multimodal Dialogue State Tracking (MM-DST) …

Content truncated to 500 words