JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12037 - 12051 December 7-11, 2022 ©2022 Association for Computational Linguistics JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization Nan Zhao∗, Haoran Li∗, Youzheng Wu, Xiaodong He JD AI Research {zhaonan8, lihaoran24}@jd.com Abstract The popularity of multimodal dialogue has stimulated the need for a new generation of dialogue agents with multimodal interactivity. When users communicate with customer ser- vice, they may express their requirements by means of text, images, or even videos. Visual information usually acts as discriminators for product models, or indicators of product fail- ures, which play an important role in the E- commerce scenario. On the other hand, de- tailed information provided by the images is limited, and typically, customer service sys- tems cannot understand the intent of users with- out the input text. Thus, bridging the gap be- tween the image and text is crucial for com- municating with customers. In this paper, we construct JDDC 2.1, a large-scale multimodal multi-turn dialogue dataset collected from a mainstream Chinese E-commerce platform1, containing about 246K dialogue sessions, 3M utterances, and 507K images, along with prod- uct knowledge bases and image category anno- tations. Over our dataset, we jointly define four tasks: the multimodal dialogue response gener- ation task, the multimodal query rewriting task, the multimodal dialogue discourse parsing task, and the multimodal dialogue summarization task. JDDC 2.1 is the first corpus with anno- tations for all the above tasks over the same dialogue sessions, which facilitates the com- prehensive research around the dialogue. In addition, we present several text-only and mul- timodal baselines and show the importance of visual information for these tasks. Our dataset and implements will be publicly available. 1 Introduction With the development of the Internet, multimodal dialogue has become much more natural and preva- lent in many scenarios, such as e-commerce, restau- rant, travel, and so on, which stimulates research ∗Equal contribution. 1https://JD.com on dialogue agents with multimodal perceptions, understanding the interaction between vision and language. Take the scene of e-commerce as an example, when users resort to customer service for solving the difficulties they encounter in the process of online shopping, there could be various forms of information, including text, images, or even videos, which establishes a great challenge for customer service systems to understand users’ requirements. Generally, users tend to express their needs with text. While sometimes, the text fails to convey enough information, and in that case, users may upload some images. For example, in Fig- ure 1, images are used for distinguishing different product models for the same brand or used for iden- tifying the location and cause of product failures. Therefore, there is an urgent need for customer ser- vice systems to understand multimodal information sent by users to provide proper responses. Multimodal information processing has been widely explored recently. Researches on image captioning (Xu et al., 2015; Anderson et al., 2018; Pan et al., …

Content truncated to 500 words