Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2243–2248, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems Ryuichiro Higashinaka1, Masahiro Mizukami2, Kotaro Funakoshi3, Masahiro Araki4, Hiroshi Tsukahara5, Yuka Kobayashi6 1NTT Corporation, 2Nara Institute of Science and Technology 3Honda Research Institute Japan, 4Kyoto Institute of Technology 5Denso IT Laboratory, Inc., 6Toshiba Corporation Abstract This paper aims to find errors that lead to dialogue breakdowns in chat-oriented dia- logue systems. We collected chat dialogue data, annotated them with dialogue break- down labels, and collected comments de- scribing the error that led to the break- down. By mining the comments, we first identified error types. Then, we calculated the correlation between an error type and the degree of dialogue breakdown it in- curred, quantifying its impact on dialogue breakdown. This is the first study to quan- titatively analyze error types and their ef- fect in chat-oriented dialogue systems. 1 Introduction Chat-oriented or open-domain dialogue systems have recently been attracting attention from so- cial and entertainment aspects (Bickmore and Cas- sell, 2001; Banchs and Li, 2012; Wilcock and Jokinen, 2013). However, since they need to deal with open-domain utterances, which current natural language processing techniques are not mature enough to handle appropriately, the sys- tem inevitably makes errors. This discourages users from talking to the system, leading to di- alogue breakdowns in conversation (Martinovsky and Traum, 2003). Here, dialogue breakdowns de- note points in dialogue where users are unable to continue the conversation. This paper aims to find errors that lead to dia- logue breakdowns in chat-oriented dialogue sys- tems. Our approach is two-fold: (1) identify error types in chat-oriented dialogue systems, and (2) calculate their impact on dialogue breakdown. For (1), we first collect chat dialogues between an au- tomated system and users, annotate the dialogues with dialogue breakdown labels, and collect com- ments that describe the error that led to the break- downs. After that, we apply automatic cluster- ing methods to the comments to obtain clusters of comments, each of which corresponds to a partic- ular error type. For (2), we calculate the correla- tion between an error type and the degree of dia- logue breakdown it incurs. This way, we can quan- tify the impact of an error type on dialogue break- down. By our approach, we hope to distinguish fatal errors from non-fatal ones, providing useful guidelines for dialogue system developers to avoid breakdowns in chat-oriented dialogue systems. 2 Data Collection For data collection, we asked dialogue researchers and their collaborators in Japan to use our chat- oriented dialogue system. The system is text- based and enables chatting in Japanese. It was built by wrapping a chat API provided by NTT DOCOMO (Onishi and Yoshimura, 2014). Since the API works on the basis of (Higashinaka et al., 2014), which uses a number of natural language processing techniques to understand and generate utterances, we expected to obtain a wide …

Content truncated to 500 words