ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7579 - 7590 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments Ruolan Yang1, Zitong Li1, Haifeng Tang2, Kenny Q. Zhu3∗ 1Shanghai Jiao Tong University, Shanghai, China 2China Merchants Bank Credit Card Center, Shanghai, China 1{wuyanzu2333, AutSky_JadeK}@sjtu.edu.cn 2thfeng@cmbchina.com 3kzhu@cs.sjtu.edu.cn Abstract Existing automatic evaluation systems of chat- bots mostly rely on static chat scripts as ground truth, which is hard to obtain, and requires access to the models of the bots as a form of “white-box testing”. Interactive evalua- tion mitigates this problem but requires hu- man involvement. In our work, we propose an interactive chatbot evaluation framework in which chatbots compete with each other like in a sports tournament, using flexible scoring metrics. This framework can efficiently rank chatbots independently from their model archi- tectures and the domains for which they are trained. 1 Introduction Evaluation of dialogue systems is an open problem. Existing automatic evaluation metrics for chitchat systems are similar to those for other text genera- tion tasks (e.g., machine translation (Papineni et al., 2002), question-answering (Rajpurkar et al., 2016), summarization (Lin, 2004)), which depends on cal- culating word overlaps with reference responses. However, for chitchats, there are usually many al- ternative but plausible responses given a situation, perhaps more than any other text generation task mentioned above. A limited number of reference responses are not sufficient to determine how good a generated response is. Moreover, such static settings are not good at assessing an interactive, context-sensitive system. Interactive human evaluation metrics usually in- volve a Likert scale evaluation after a multi-turn conversation with the bot to be assessed. While this method is a step up from the previous static evalua- tion, it is difficult for human judges to give a con- crete score to any bot. Comparing the performance of two bots is easier. Thus ACUTE-EVAL (Li et al., ∗ Kenny Q. Zhu is the corresponding author, and is par- tially supported by NSFC Grant No. 91646205, and SJTU- CMBCC Joint Research Scheme. 2019) asks the judges to make a binary judgment of who is better in conversations between two iden- tical bots or between a human and a bot. A more advanced version of that is Spot The Bot (Deriu et al., 2020) which models the human evaluation of a conversation after the Turing test. However, such a process is still time-consuming and costly, compared with automatic evaluations. In our opinion, a good method for evaluating multi-turn conversational model/system should sat- isfy the following requirements: i) be as efficient and inexpensive as possible; ii) can truly reflect a model’s ability to conduct a human conversation; iii) evaluation results should correlate well with human judgments; iv) can be used to compare and rank the capabilities of a set of models/systems. Toward that goal, in this work, we propose an automatic interactive evaluation framework, which is called ChatMatch(CM) for chitchat agents. This framework can be …

Content truncated to 500 words