Knowledge Grounded Multimodal Dialog Generation in Task-oriented Settings

Knowledge Grounded Multimodal Dialog Generation in Task-oriented Settings Deeksha Varshney, Anushkha Singh, Asif Ekbal Department of Computer Science and Engineering Indian Institute of Technology Patna {1821cs13, anushkha 1901me72, asif}@iitp.ac.in Abstract Knowledge-grounded dialogue generation is the process of formulating an informed re- sponse based on both the conversation con- text and external knowledge. Multi-modality in dialogue systems has paved the way for more robust conversational bots. Any multi- modal system seeks to bridge the gap between language and vision by combining informa- tion from image, audio, video, and text, all of which are often complementary. In every task-oriented dialog system, product or ser- vice attributes are crucial for satisfying the user’s needs. We propose the task of knowl- edge grounded response generation in a multi- modal task-oriented dialog setting. We em- ploy a multi-modal knowledge grounded in- cremental transformer network to generate re- sponses that combine information from con- text, visuals, and external knowledge. We prepare a Knowledge Grounded Multi-Modal Dialog (KGMMD) dataset to help users in more efficient decision making. Dialog ut- terances, accompanying images, and knowl- edge in the form of hotel reviews from the hotel domain are all included in the dataset. The newly formed KGMMD dataset has been subjected to both quantitative and qualitative analysis. Evaluation results on the KGMMD datasets show that the proposed methodology outperforms the baseline models for knowl- edge grounded multimodal response genera- tion. 1 Introduction Conversational agents now have more possibilities thanks to advances in artificial intelligence (AI). Human-machine interaction is an important appli- cation of artificial intelligence that assists people in their daily lives. AI advancements have resulted in the invention of personal assistants such as Ap- ple’s Siri, Amazon’s Alexa, and Microsoft’s Cor- tana, which aid people in their daily tasks. Growing demands in numerous industries such as travel, en- tertainment, shopping, and so on necessitate conver- sational agents to communicate by merging infor- mation from various modalities to develop a robust system. A multimodal dialogue system allows for inter- disciplinary research by bridging the gap between visual and text information. Data from several di- mensions, such as text, image, audio, and video, has been shown to offer all of the details required to build effective end-to-end dialogue systems (Le et al., 2019a; Saha et al., 2018). Multimodal conver- sational systems supplement existing dialogue sys- tems by supplying information that unimodal sys- tems lack. Several recent efforts on multimodal di- alogue systems (Chauhan et al., 2019; Das et al., 2017; Le et al., 2019b) have spurred interest in this field by combining data from a variety of sources, including texts, and images. Many natural language generation (NLG) tasks require semantic comprehension, especially when it is supplemented by real-world data. Building a good robust dialog system requires integrating the neces- sary subjects, entities, and relations mentioned in the user’s statement while also providing an acceptable answer (Wu et al., 2019; Agarwal et al., 2018b). It is critical to obtain precise information about the ser- vice requested by the user in order …

Content truncated to 500 words