CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4097 - 4111 December 7-11, 2022 ©2022 Association for Computational Linguistics CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation Yinpei Dai, Wanwei He, Bowen Li, Yuchuan Wu, Zheng Cao, Zhongqi An, Jian Sun, Yongbin Li∗ Alibaba Group, Beijing, China {yinpei.dyp,hewanwei.hww,shengxiu.wyc,zhengzhi.cz, zhongqi.azq,jian.sun,shuide.lyb}@alibaba-inc.com Abstract Practical dialog systems need to deal with var- ious knowledge sources, noisy user expres- sions, and the shortage of annotated data. To better solve the above problems, we propose CGoDial1, a new challenging and compre- hensive Chinese benchmark for multi-domain Goal-oriented Dialog evaluation. It contains 96,763 dialog sessions, and 574,949 dialog turns totally, covering three datasets with differ- ent knowledge sources: 1) a slot-based dialog (SBD) dataset with table-formed knowledge, 2) a flow-based dialog (FBD) dataset with tree- formed knowledge, and a retrieval-based dialog (RBD) dataset with candidate-formed knowl- edge. To bridge the gap between academic benchmarks and spoken dialog scenarios, we ei- ther collect data from real conversations or add spoken features to existing datasets via crowd- sourcing. The proposed experimental settings include the combinations of training with either the entire training set or a few-shot training set, and testing with either the standard test set or a hard test subset, which can assess model ca- pabilities in terms of general prediction, fast adaptability and reliable robustness. 1 Introduction Goal-oriented dialog systems converse with users naturally, helping them fulfill specific goals such as restaurant booking, hotel reservation, and flight search. Many popular datasets have been intro- duced to facilitate the dialog research, ranging from simple single-domain (Wen et al., 2017) to more difficult multi-domain dialogs (Budzianowski et al., 2018). Similar to the widely studied GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) for general natural language understanding, large- scale benchmarks have been specifically designed for dialog systems. For instance, Mehri et al. (2020) ∗Corresponding author 1Dataset available at https://github.com/Alibaba Research/DAMO-ConvAI/cgodial. proposed the DialoGLUE, a collection of seven popular English dialog datasets, to assess the natu- ral language understanding ability in a few-shot set- ting. More recently, Peng et al. (2021b) introduced the RADDLE, another large-scale benchmark that includes a diagnostic checklist facilitating detailed robustness analysis for pre-trained language mod- els (Radford et al.; Peng et al., 2021a). However, compared to the abundance of English dialog benchmarks, the resource of Chinese dialog datasets in goal-oriented dialog remains quite lim- ited, let alone the large-scale benchmark suitable for the dialog evaluation of Chinese pre-trained lan- guage models. Moreover, current Chinese dialog datasets, such as CrossWOZ (Zhu et al., 2020) and RiSAWOZ (Quan et al., 2020), mainly focus on the schema-guided dialog as in the classical Multi- WOZ dataset (Budzianowski et al., 2018), where conversations are grounded on a table-formed on- tology. This kind of ontology usually defines a set of possible slot-value pairs to be recognized during the dialog for database query (Young et al., 2013). In real-world applications, however, more types of goal-oriented dialogs are involved. For instance, in Xie et …

Content truncated to 500 words