Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 462–472, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation Nasrin Mostafazadeh1*, Chris Brockett2, Bill Dolan2, Michel Galley2, Jianfeng Gao2, Georgios P. Spithourakis3*, Lucy Vanderwende2 1 BenevolentAI, 2 Microsoft Research, 3 University College London nasrin.m@benevolent.ai, chrisbkt@microsoft.com Abstract The popularity of image sharing on social media and the engagement it creates be- tween users reflect the important role that visual context plays in everyday conver- sations. We present a novel task, Image- Grounded Conversations (IGC), in which natural-sounding conversations are gener- ated about a shared image. To benchmark progress, we introduce a new multiple- reference dataset of crowd-sourced, event- centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on so- cial media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between hu- man performance and that of both neu- ral and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialog research. 1 Introduction Bringing together vision & language in one in- telligent conversational system has been one of the longest running goals in AI (Winograd, 1972). Advances in image captioning (Fang et al., 2014; Chen et al., 2015; Donahue et al., 2015) have enabled much interdisciplinary research in vision and language, from video transcription (Rohrbach et al., 2012; Venugopalan et al., 2015), to an- swering questions about images (Antol et al., 2015; Malinowski and Fritz, 2014), to storytelling around series of photographs (Huang et al., 2016). * This work was performed at Microsoft. User1: My son is ahead and surprised! User2: Did he end up winning the race? User1: Yes he won, he can’t believe it! Figure 1: A naturally-occurring Image-Grounded Conversation. Most recent work on vision & language fo- cuses on either describing (captioning) the image or answering questions about their visible con- tent. Observing how people naturally engage with one another around images in social media, it is evident that it is often in the form of conversa- tional threads. On Twitter, for example, upload- ing a photo with an accompanying tweet has be- come increasingly popular: in June 2015, 28% of tweets reportedly contained an image (Morris et al., 2016). Moreover, across social media, the conversations around shared images range beyond what is explicitly visible in the image. Figure 1 illustrates such a conversation. As this example shows, the conversation is grounded not only in the visible objects (e.g., the boys, the bikes) but more importantly, in the events and actions (e.g., the race, winning) implicit in the image that is ac- companied by the textual utterance. To humans, it is these latter aspects that are likely to be the most interesting and most meaningful components of a natural conversation, and to the systems, inferring such …

Content truncated to 500 words