Ordinal and Attribute Aware Response Generation in a Multimodal Dialogue System

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5437–5447 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5437 Ordinal and Attribute Aware Response Generation in a Multimodal Dialogue System Hardik Chauhan1∗, Mauajama Firdaus2∗, Asif Ekbal2, Pushpak Bhattacharyya2 1 Department of Electrical Engineering, Indian Institute of Technology Roorkee, India 2 Department of Computer Science and Engineering, Indian Institute of Technology Patna, India (chauhanhardik23@gmail.com),(mauajama.pcs16,asif,pb)@iitp.ac.in Abstract Multimodal dialogue systems have opened new frontiers in the traditional goal-oriented dialogue systems. The state-of-the-art dia- logue systems are primarily based on uni- modal sources, predominantly the text, and hence cannot capture the information present in the other sources such as videos, audios, im- ages etc. With the availability of large scale multimodal dialogue dataset (MMD) (Saha et al., 2018) on the fashion domain, the visual appearance of the products is essential for un- derstanding the intention of the user. With- out capturing the information from both the text and image, the system will be incapable of generating correct and desirable responses. In this paper, we propose a novel position and attribute aware attention mechanism to learn enhanced image representation conditioned on the user utterance. Our evaluation shows that the proposed model can generate appropriate responses while preserving the position and at- tribute information. Experimental results also prove that our proposed approach attains su- perior performance compared to the baseline models, and outperforms the state-of-the-art approaches on text similarity based evaluation metrics. 1 Introduction With the advancement in Artificial Intelligence (AI), dialogue systems have become a prominent part in today’s virtual assistant, which helps users to converse naturally with the system for effective task completion. Dialogue systems focus on two broad categories - open domain conversations with casual chit chat and goal-oriented systems where the system is designed to solve a particular task for the user belonging to a specific domain. Re- sponse generation is a crucial component of every conversational agent. The task of “how to say” ∗ First two authors are jointly the first authors the information to the user is the primary objec- tive of every response generation module. One of the running goals of AI is to bring language and vision together in building robust dialogue systems. Advances in visual question answer- ing (VQA) (Kim et al., 2016; Xiong et al., 2016; Ben-Younes et al., 2017), and image captioning (Anderson et al., 2018; Chen et al., 2018) have ensured interdisciplinary research in natural lan- guage processing (NLP) and computer vision. Re- cently, several works in dialogue systems incorpo- rating both vision and language (Das et al., 2017a; Mostafazadeh et al., 2017) have shown promising research directions. Goal oriented dialogue systems are majorly based on textual data (unimodal source). With in- creasing demands in the domains like retail, travel, entertainment, conversational agents that can con- verse by combining different modalities is an es- sential requirement for building the robust sys- tems. Knowledge from different modalities car- ries complementary information about the vari- ous aspects …

Content truncated to 500 words