Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3971–3984, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3971 Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems Jan Deriu1, Don Tuggener1, Pius von D¨aniken1, Jon Ander Campos3, Alvaro Rodrigo2, Thiziri Belkacem4, Aitor Soroa3, Eneko Agirre3, and Mark Cieliebak1 1Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland, {deri, tuge, vode, ciel}@zhaw.ch 2National Distance Education University (UNED), Madrid, Spain, alvarory@lsi.uned.es 3University of the Basque Country (UPV/EHU), Donostia, Spain, {jonander.campos, a.soroa, e.agirre}@ehu.eus 4Synapse D´eveloppement, Toulouse, France, belkacemthiziri@gmail.com Abstract The lack of time-efficient and reliable evalua- tion methods hamper the development of con- versational dialogue systems (chatbots). Eval- uations requiring humans to converse with chatbots are time and cost-intensive, put high cognitive demands on the human judges, and yield low-quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only anno- tate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversa- tions). These annotations then allow us to rank chatbots regarding their ability to mimic the conversational behavior of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chatbot can uphold human-like behav- ior the longest, i.e., Survival Analysis. This metric has the ability to correlate a bot’s per- formance to certain of its characteristics (e.g., fluency or sensibleness), yielding interpretable results. The comparably low cost of our frame- work allows for frequent evaluations of chat- bots during their evaluation cycle. We empiri- cally validate our claims by applying Spot The Bot to three domains, evaluating several state- of-the-art chatbots, and drawing comparisons to related work. The framework is released as a ready-to-use tool. 1 Introduction Evaluation is a long-standing issue in developing conversational dialogue systems (i.e., chatbots). The underlying difficulty in evaluation lies in the problem’s open-ended nature, as chatbots do not solve a clearly-defined task whose success can be measured in relation to an a priori defined ground truth. Automatic metrics have so far failed to show high correlation with human evaluations (Liu et al., 2016; Lowe et al., 2017; Mehri and Eskenazi, 2020). Human evaluation approaches are mainly classified according to the following: single-turn vs. multi-turn evaluation, and direct user evalua- tion vs. expert judge evaluation. Single-turn anal- ysis is usually performed by a human judge that rates a single response of the bot to a given con- text, whereas multi-turn analysis is often performed by a user that interacts with the bot and rates the interaction. Single-turn ratings disregard the multi- turn nature of a dialogue (See et al., 2019). Al- though more and more multi-turn evaluations are performed, most of them are based on human-bot conversations, which are costly to obtain and tend to suffer from low quality (Dinan et al., 2020a). The instructions to be followed by …

Content truncated to 500 words