MMChat: Multi-Modal Chat Dataset on Social Media

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5778–5786 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5778 MMChat: Multi-Modal Chat Dataset on Social Media Yinhe Zheng♠∗, Guanyi Chen♣∗, Xin Liu♡, Jian Sun♠† ♠ Alibaba Group, ♣ Utrecht University, ♡Samsung Research China - Beijing (SRC-B) zhengyinhe1@163.com, g.chen@uu.nl, jian.sun@alibaba-inc.com Abstract Incorporating multi-modal contexts in conversation is important for developing more engaging dialogue systems. In this work, we explore this direction by introducing MMCHAT: a large-scale Chinese multi-modal dialogue corpus (32.4M raw dialogues and 120.84K filtered dialogues). Unlike previous corpora that are crowd-sourced or collected from fictitious movies, MMCHAT contains image-grounded dialogues collected from real conversations on social media, in which the sparsity issue is observed. Specifically, image-initiated dialogues in common communications may deviate to some non-image-grounded topics as the conversation proceeds. To better investigate this issue, we manually annotate 100K dialogues from MMCHAT and further filter the corpus accordingly, which yields MMCHAT-HF. We develop a benchmark model to address the sparsity issue in dialogue generation tasks by adapting the attention routing mechanism on image features. Experiments demonstrate the usefulness of incorporating image features and the effectiveness of handling the sparsity of image features. Keywords: Dialogue Systems, Multi-Modality, Social Media 1. Introduction The ability to converse like a human is one of the desiderata for building open-domain dialogue sys- tems (Adiwardana et al., 2020). Current attempts to build human-like open-domain dialogue systems gen- erally follow two angles: 1) Enriching the dialogue sys- tem with textual or structural contexts such as knowl- edge (Madotto et al., 2018) or personalities (Zhang et al., 2018a; Zheng et al., 2019); 2) Enabling the dia- logue systems to perceive multi-modality contexts be- yond text such as vision, voice, or even gesture (Shuster et al., 2020b; Shuster et al., 2020c; Liao et al., 2018; Ju et al., 2019). Systems built following the second an- gle are also known as Multi-Modal Dialogue Systems (MMDSs). To facilitate the development of data-driven MMDSs, a few dialogue datasets containing visual information have been constructed (Mostafazadeh et al., 2017; Mo- gadala et al., 2019; AlAmri et al., 2019; Kottur et al., 2019; Pasunuru and Bansal, 2018; Shuster et al., 2020a; Meng et al., 2020). For instance, Shuster et al. (2020a) introduced a crowd-sourced image grounded dialogue corpus IMAGE-CHAT, in which annotators are employed to chat in accordance with given im- ages. Meng et al. (2020) proposed OPENVIDIAL by directly extracting dialogues and their visual contexts from movies and TV series. There are also works on visual question answering (Das et al., 2017) that fo- cus on the question answering tasks involving image inputs. A significant drawback of existing datasets is that they postulated every utterance in a dialogue to be grounded on the given image. Nevertheless, this is not always true in our daily communications. Concretely, the topic triggered by the image may drift in the conversation flow so that not every utterance in a dialogue session is ∗ Equal Contribution † Corresponding …

Content truncated to 500 words