Don’t be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2357–2367 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2357 Don’t be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System Libo Qin, Tianbao Xie, Shijue Huang, Qiguang Chen, Xiao Xu, Wanxiang Che∗ Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China {lbqin,tianbaoxie,sjhuang,qgchen,xxu,car}@ir.hit.edu.cn Abstract Consistency Identification has obtained re- markable success on open-domain dialogue, which can be used for preventing inconsis- tent response generation. However, in con- trast to the rapid development in open-domain dialogue, few efforts have been made to the task-oriented dialogue direction. In this pa- per, we argue that consistency problem is more urgent in task-oriented domain. To facilitate the research, we introduce CI-ToD, a novel dataset for Consistency Identification in Task- oriented Dialog system. In addition, we not only annotate the single label to enable the model to judge whether the system response is contradictory, but also provide more fine- grained labels (i.e., Dialogue History Inconsis- tency, User Query Inconsistency and Knowl- edge Base Inconsistency) to encourage model to know what inconsistent sources lead to it. Empirical results show that state-of-the-art methods only achieve 51.3%, which is far be- hind the human performance of 93.2%, indi- cating that there is ample room for improv- ing consistency identification ability. Finally, we conduct exhaustive experiments and qual- itative analysis to comprehend key challenges and provide guidance for future directions. All datasets and models are publicly available at https://github.com/yizhen20133868/CI-ToD. 1 Introduction Task-oriented dialogue systems (ToDs) (Young et al., 2013) aim to achieve user goals such as ho- tel booking and restaurant reservation, has gained more attention recently in both academia and in- dustries. Over the last few years, two promising research directions in ToDs have emerged. The first focuses on a pipeline approach, which consists of modularly connected components (Wu et al., 2019a; Takanobu et al., 2020; Peng et al., 2020; Li et al., 2020). The second direction employs an end- ∗Email corresponding. User: Give me directions to the closest grocery store. System: There is a whole foods 2 miles away and their address is 880_ames_ct. User: I need a route that avoids all heavy traffic. System: mandarin roots is located at 271 springer street. Dialog System poi … address whole foods … 880_ames_ct mandarin roots … 271 springer street Inconsistent Figure 1: A system response generation example by the state-of-the-art end-to-end task-oriented dialogue model DF-Net (Qin et al., 2020b). to-end model, which directly takes the sequence-to- sequence (Seq2Seq) model to generate a response from a dialogue history and a corresponding knowl- edge base (KB) (Eric et al., 2017; Madotto et al., 2018; Wen et al., 2018; Qin et al., 2019b; Wu et al., 2019b; Qin et al., 2020b) In recent years, with the burst of deep neural networks and the evolution of pre-trained language models, the research of ToDs has obtained great success. While the success is indisputable, previ- ous work have shown that it’s inevitable to gen- erate …

Content truncated to 500 words