XDailyDialog: A Multilingual Parallel Dialogue Corpus

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12240–12253 July 9-14, 2023 ©2023 Association for Computational Linguistics XDailyDialog: A Multilingual Parallel Dialogue Corpus Zeming Liu1∗, Ping Nie2∗, Jie Cai2∗, Haifeng Wang3†, Zheng-Yu Niu3, Peng Zhang4, Mrinmaya Sachan5, Kaiping Peng4 1Research Center for Social Computing and Information Retrieval, 2Peking UniversityHarbin Institute of Technology, Harbin, China 3Baidu Inc., Beijing, China 4Tsinghua University zmliu@ir.hit.edu.cn {ping.nie, caijie}@pku.edu.cn5ETH Zurich {wanghaifeng, niuzhengyu}@baidu.com Abstract High-quality corpora are significant to the de- velopment of dialogue models. However, most existing corpora for open-domain dialogue modeling are limited to a single language. The absence of multilingual open-domain dialog corpora not only limits the research on mul- tilingual or cross-lingual transfer learning but also hinders the development of robust open- domain dialogue systems that can be deployed in other parts of the world. In this paper, we provide a multilingual parallel open-domain dialog dataset, XDailyDialog,1 to enable re- searchers to explore the challenging task of multilingual and cross-lingual open-domain di- alogue. XDailyDialog includes 13K dialogues aligned across 4 languages (52K dialogues and 410K utterances in total). We then propose a di- alogue generation model, kNN-Chat, which has a novel kNN-search mechanism to sup- port unified response retrieval for monolingual, multilingual, and cross-lingual dialogue. Ex- periment results show the effectiveness of this framework. 1 Introduction Developing high-quality open-domain dialogue systems is one of the key challenges in Artificial In- telligence. Unlike closed-domain dialogue systems which deal with specific kinds of conversations (like a chatbot for customer support), open-domain dialogue systems can engage in conversation on any topic. In recent years, there has been a signifi- cant increase in research on dialogue due to the rise of voice-based bots, such as Meena (Adiwardana et al., 2020), BlenderBot (Roller et al., 2021) and XiaoIce (Zhou et al., 2020). To advance the qual- ity of open-domain dialogue systems, many large- scale corpora have been created (Sordoni et al., 2015; See et al., 2019; Yang et al., 2018; Mazaré et al., 2018; Keskar et al., 2019). ∗ Equal contribution † Corresponding author: Haifeng Wang. 1https://github.com/liuzeming01/XDailyDialog Monolingual Dialog … A: Can I have the check ? B: Right away, Sir. … … A:请给我支票好吗？ B:马上，先生。 … … … … … A: Can I have the check ? B: 马上，先生。 … … … A: Can I have the check ? B: Subito, Sir. … … A: 请给我支票好吗？ B: Right away, Sir. … En Zh De En En De Zh En En En Zh Zh De De It It … A: Can I have the check ? A: 请给我支票好吗？ A: Kann ich bitte den Scheck haben? A: Posso avere il conto, per favore … … B: Right away, Sir. B:马上，先生。 B: Subito, Sir. B: Va bene, signore. … En Zh De It Ar Cs De En Es Et Fi Fr It Ja Ko Ni Ro Ru Vi Zh Crosslingual Dialog Multilingual Dialog … B: Right away, Sir. A: Kann ich bitte den Scheck haben? A: Kann ich bitte den Scheck haben? …

Content truncated to 500 words