MultiDM-GCN: Aspect-guided Response Generation in Multi-domain Multi-modal Dialogue System using Graph Convolutional Network

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2318–2328 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2318 MultiDM-GCN: Aspect-guided Response Generation in Multi-domain Multi-modal Dialogue System using Graph Convolutional Network Mauajama Firdaus∗, Nidhi Thakur∗, Asif Ekbal Department of Computer Science and Engineering Indian Institute of Technology Patna Patna, India (mauajama.pcs16,asif)@iitp.ac.in Abstract In the recent past, dialogue systems have gained immense popularity and have become ubiquitous. During conversations, humans not only rely on languages but seek contextual in- formation through visual contents as well. In every task-oriented dialogue system, the user is guided by the different aspects of a product or service that regulates the conversation to- wards selecting the product or service. In this work, we present a multi-modal conversational framework for a task-oriented dialogue setup that generates the responses following the dif- ferent aspects of a product or service to cater to the user’s needs. We show that the responses guided by the aspect information provide more interactive and informative responses for bet- ter communication between the agent and the user. We first create a Multi-domain Multi- modal Dialogue (MDMMD) dataset having conversations involving both text and images belonging to the three different domains, such as restaurants, electronics, and furniture. We implement a Graph Convolutional Network (GCN) based framework that generates appro- priate textual responses from the multi-modal inputs. The multi-modal information having both textual and image representation is fed to the decoder and the aspect information for generating aspect guided responses. Quantita- tive and qualitative analyses show that the pro- posed methodology outperforms several base- lines for the proposed task of aspect-guided re- sponse generation. 1 Introduction Conversational systems have become ubiquitous in our everyday lives. Previous research suggests that the conversational agents need to be more interac- tive and informative for building engaging systems (Takayama and Arase, 2019; Shukla et al., 2019). ∗ First two authors have contributed equally These research indicates that engaging conversa- tions include visual cues (e.g., a video or images) or audio cues (e.g., tone, the pitch of the speaker). Information contained in these cues is often inte- gral for the conversation. In Figure 1, we show an example of a conversation where the visual cues in the form of images are crucial for better un- derstanding and interactive dialogue between the agent and the user. The appropriate responses to the user queries are highly dependent on the visual information pertaining to the different aspects of the various images in the conversation. Thus, it is natural to conclude that a conversational agent would be more effective if the visual information were part of its underlying conversational model. Multi-modality in goal-oriented dialogue systems Figure 1: Examples from the Multi-domain Multi- modal Dialogue(MDMMD) dataset (Saha et al., 2018) for the fashion domain has estab- lished the significance of visual information for ef- fective communication between the user and agent. Inspired by their works, we take a step forward by creating a multi-modal aspect guided response framework for a multi-domain goal-oriented …

Content truncated to 500 words