The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics

The Dialogue Breakdown Detection Challenge: Task Description, Datasets, and Evaluation Metrics Ryuichiro Higashinaka1, Kotaro Funakoshi2 Yuka Kobayashi3, Michimasa Inaba4 1NTT Media Intelligence Laboratories 2Honda Research Institute Japan Co., Ltd. 3Toshiba Corporation 4Hiroshima City University Abstract Dialogue breakdown detection is a promising technique in dialogue systems. To promote the research and development of such a technique, we organized a dialogue breakdown detection challenge where the task is to detect a system’s inappropriate utterances that lead to dialogue breakdowns in chat. This paper describes the design, datasets, and evaluation metrics for the challenge as well as the methods and results of the submitted runs of the participants. Keywords: dialogue breakdown, chat-oriented dialogue, evaluation workshop 1. Introduction Although voice agent services are beginning to appear on the market, the limited capabilities of these systems mean that humans and machines still cannot converse as naturally as two humans. The main problem is that systems typically make inappropriate utterances that lead to dialogue break- downs. By dialogue breakdown, we mean a situation in a dialogue where users cannot proceed with the conversation (Martinovsky and Traum, 2003). To avoid this situation, technology for dialogue breakdown detection is essential because such technology will enable systems to avoid the creation of inappropriate utterances and also to identify di- alogue breakdowns when they occur and perform the nec- essary recovery procedures. This paper reports the results of our “Dialogue breakdown detection challenge”, which is an evaluation workshop ded- icated to dialogue breakdown detection. The aim of the challenge was to bring together various methods for di- alogue breakdown detection and discuss possible evalua- tion metrics. Although there have been other dialogue- related evaluation campaigns, including a spoken dialogue challenge (Black and Eskenazi, 2009) and a dialogue state tracking challenge (Williams et al., 2013), our challenge is unique in that we deal with chat dialogues. We focus on chat because, compared to the errors in task-oriented dialogues (Bear et al., 1992; Carpenter et al., 2001; Bu- lyko et al., 2005), we can observe more diverse dialogue breakdown phenomena (Higashinaka et al., 2015a), which makes it possible to develop general and robust techniques for breakdown detection. Our challenge is also unique in that we deal with a highly subjective phenomenon, posing difficulties in creating appropriate evaluation metrics. In this paper, we present how we designed the task, pre- pared the datasets, and conceived the evaluation metrics for the challenge. The event was held in October 2015 and there were six participants. We also report the detection methods submitted by the participants and provide their re- sults. We finish with a discussion of the validity of the eval- uation metrics we utilized. 2. Task description The task of the dialogue breakdown detection challenge is to detect whether the system utterance causes dialogue breakdown in a given dialogue context. For example, con- sider the following piece of dialogue: S1: It’s nice to go shopping alone. U1: I agree. That’s nice. S2: Shopping takes time. U2: Window shopping is also fun. S3: It’s fun …

Content truncated to 500 words