Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1057–1067 August 1–6, 2021. ©2021 Association for Computational Linguistics 1057 Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency Zekang Li12, Jinchao Zhang3, Zhengcong Fei12, Yang Feng12∗, Jie Zhou3 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences 3 Pattern Recognition Center, WeChat AI, Tencent Inc, China {lizekang19g,feizhengcong,fengyang}@ict.ac.cn {dayerzhang,withtomzhou}@tencent.com Abstract A good open-domain chatbot should avoid pre- senting contradictory responses about facts or opinions in a conversational session, known as its consistency capacity. However, evalu- ating the consistency capacity of a chatbot is still challenging. Employing human judges to interact with chatbots on purpose to check their capacities is costly and low-efficient, and difficult to get rid of subjective bias. In this paper, we propose the Addressing Inquiries about History (AIH), an efficient and practi- cal framework for the consistency evaluation. At the conversation stage, AIH attempts to ad- dress appropriate inquiries about the dialogue history to induce the chatbot to redeclare the historical facts or opinions. We carry out the conversation between chatbots, which is more efficient than the human-bot interaction and can also alleviate the subjective bias. In this way, we manage to rapidly obtain a dialog session that contains responses with high con- tradiction possibilities. At the contradiction recognition stage, we can either employ hu- man judges or a natural language inference (NLI) model to recognize whether the answers to the inquiries are contradictory with history. Finally, we are able to rank chatbots according to the contradiction statistics. Experiments on open-domain chatbots show that our approach can efficiently and reliably assess the consis- tency capacity of chatbots and achieve a high ranking correlation with the human evaluation. We release the framework and hope to help im- prove the consistency capacity of chatbots.1 1 Introduction In recent years, open-domain intelligent chatbots (Adiwardana et al., 2020b; Bao et al., 2020; Smith ∗Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc. Yang Feng is the corresponding author. Work was done when Zekang Li and Zhengcong Fei were intern at WeChat AI. 1https://github.com/ictnlp/AIH DialoGPT 762M Human: What movies do you like most? DialoGPT: The Unforgiven from Metallica. Human: What do you think of the Unforgiven? DialoGPT: I never heard of it. Blender 2.7B Human: What do you like to cook? Blender: I only cook Indian cuisine. How about you? Human: I enjoy cooking Chinese food, especially the dumplings. Blender: Chinese food is delicious. I also like cooking the Chinese food. Plato 1.6B Human: Do you like dogs? Plato: Yes, dogs are cool, I prefer cats though. Human: Do you have a dog? Plato: I do not, but I’d love a dog. I think dogs are so cute. Human: I love dogs very much. I have three dogs. Plato: I have three dogs too. One puppy and two Huskies. Table 1: Several human-bot conversations demonstrate that popular chatbots (DialoGPT, Blender, and Plato) …

Content truncated to 500 words