MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7348–7363 July 9-14, 2023 ©2023 Association for Computational Linguistics MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation Jiazhan Feng1∗ Qingfeng Sun2 Can Xu2 Pu Zhao2 Yaming Yang2 Chongyang Tao2 Dongyan Zhao1,3 Qingwei Lin2 1Wangxuan Institute of Computer Technology, Peking University 2Microsoft Corporation 3National Key Laboratory of General Artificial Intelligence, BIGAI {fengjiazhan,zhaody}@pku.edu.cn {qins,caxu,puzhao,yayaming,chotao,qlin}@microsoft.com Abstract Responding with multi-modal content has been recognized as an essential capability for an in- telligent conversational agent. In this paper, we introduce the MMDialog dataset to facili- tate multi-modal conversation better. MMDia- log is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 88x. Second, it contains massive topics to generalize the open domain. To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios. In addition, we build two baselines for the above tasks with state-of-the- art techniques and report their experimental performance. We also propose a novel eval- uation metric MM-Relevance to measure the multi-modal responses. Our dataset is avail- able in https://github.com/victorsungo/ MMDialog. 1 Introduction Empowering machines to converse like humans is a long-cherished goal of AI community, and there is growing interest in developing open-domain con- versational agents (Li et al., 2017a; Gao et al., 2018; Ghazvininejad et al., 2018; Zhou et al., 2018a; Ope- nAI, 2022; Thoppilan et al., 2022). To usher ma- chines into the real visual physical world of human, it is a desirable trait of conversational agents to understand, perceive, and respond appropriately to multi-modality contexts beyond pure text (Das et al., 2017; Mostafazadeh et al., 2017; Shuster Asia.∗ Work done during the internship at Microsoft Research Corresponding author: caxu@microsoft.com and puzhao@microsoft.com At the base of a muddy ditch is the first primrose of my spring - glowing in the grey, a little spot of hope, brave, beautiful and perfect. Hi Chris, wow well spotted with the beautiful flower, I love walking alongside the river where there is a bluebell way Love every photo. Especially the weeping willow. Thanks. It’s nice to enjoy the wildlife nature and walk all the way to Winchester’s great scenery. I live in Scotland. We have woods opposite with bluebells but not as thick as yours, but have a river with kingfisher, Heron and dipper. I would love to visit Scotland and witness the beautiful hills seas, and wildlife nature. We have a lots of cheeky squirrels and wood pigeons, I love to listen them in morning. That's an amazing story thanks for sharing. Figure 1: An example of human conversation in our MMDialog dataset. They are talking about scenery and wildlife with both text and various images. et al., 2020; Sun et al., 2022), which is similar to communicating through messenger tools …

Content truncated to 500 words