Image-Chat: Engaging Grounded Conversations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2414–2429 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2414 Image-Chat: Engaging Grounded Conversations Kurt Shuster, Samuel Humeau, Antoine Bordes, Jason Weston Facebook AI Research {kshuster,samuelhumeau,abordes,jase}@fb.com Abstract To achieve the long-term goal of machines be- ing able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art im- age and text representations, considering var- ious ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emo- tional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and hu- man evaluations of engagingness show the ef- ficacy of our approach; in particular, we ob- tain state-of-the-art performance on the exist- ing IGC task, and our best performing model is almost on par with humans on the Image- Chat test set (preferred 47.7% of the time). 1 Introduction A key way for machines to exhibit intelligence is for them to be able to perceive the world around them – and to be able to communicate with humans in natural language about that world. To speak natu- rally with humans it is necessary to understand the natural things that humans say about the world they live in, and to respond in kind. This involves under- standing what they perceive, e.g. the images they see, what those images mean semantically for hu- mans, and how mood and style shapes the language and conversations derived from these observations. In this work we take a step towards these goals by considering grounded dialogue involving open- ended discussion of a given image, a setting that is naturally fun for humans (Hu et al., 2014), and study neural conversational models for task. In par- ticular, we explore both generative and retrieval models that handle multimodal dialogue by fusing Transformer architectures (Vaswani et al., 2017) for encoding dialogue history and responses and ResNet architectures (He et al., 2016) for encoding images. We propose ways to fuse those modalities together and perform a detailed study including both automatic evaluations, ablations and human evaluations of our models using crowdworkers. To train and evaluate such models, we collect a large set of human-human crowdworker conversa- tions, with the aim of training a model to engage a human in a similar fashion, consisting of 202k diverse images and 401k utterances over the im- ages, with 215 different style traits (e.g., optimistic, skeptical or frivolous) to promote engaging conver- sation. The dataset is …

Content truncated to 500 words