Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15044–15071 July 9-14, 2023 ©2023 Association for Computational Linguistics Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems Sarah E. Finch∗and James D. Finch∗and Jinho D. Choi Department of Computer Science Emory University Atlanta, GA, USA {sfillwo, jdfinch, jinho.choi}@emory.edu Abstract Despite tremendous advancements in dialogue systems, stable evaluation still requires human judgments producing notoriously high-variance metrics due to their inherent subjectivity. More- over, methods and labels in dialogue evaluation are not fully standardized, especially for open- domain chats, with a lack of work to compare and assess the validity of those approaches. The use of inconsistent evaluation can misinform the performance of a dialogue system, which becomes a major hurdle to enhance it. Thus, a dimensional evaluation of chat-oriented open- domain dialogue systems that reliably measures several aspects of dialogue capabilities is de- sired. This paper presents a novel human eval- uation method to estimate the rates of many dialogue system behaviors. Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches. The analysis demonstrates that our behavior method is more suitable than alterna- tive Likert-style or comparative approaches for dimensional evaluation of these systems. 1 Introduction Recent work in human-computer chat has made remarkable progress. Multi-turn open-domain (MTOD) models are capable of holding engaging conversations with humans (Roller et al., 2021; Adiwardana et al., 2020). However, there remain a number of challenges facing MTOD chatbots such as hallucinations (Shuster et al., 2021), common- sense violations (Zhou et al., 2021), and consis- tency issues (Nie et al., 2021). A significant obsta- cle for research that addresses these challenges is the difficulty in formulating an appropriate evalua- tion methodology due to the inherent subjectivity in determining chat quality (van Miltenburg et al., 2021). Since existing automatic evaluation metrics have been shown to be biased measures of chat quality (Liu et al., 2016; Sai et al., 2019; Deriu *Contributed equally to this work as first authors. et al., 2022), evaluation using human judgments is standard, although the type of human judgments varies widely across works (Finch and Choi, 2020). Overall, there are few works comparing and as- sessing the validity of various human evaluation methods. The result of this gap in the literature is that the relative sensitivity, interpretability, and importance of the metrics used to evaluate chat models are not well understood. A dimensional approach for evaluating chat models that measures different aspects of chat quality would surely aid progress (van Miltenburg et al., 2021). However, to our knowledge, no work has investigated the cover- age of a comprehensive set of evaluation metrics. Consequently, existing chat model evaluation re- sults provide an incomplete picture of the strengths and weaknesses of MTOD chatbots. This paper ad- dresses these limitations of previous work through the following three contributions: 1. A novel, dimensional human evaluation method that measures the rate of chatbot behaviors im- pacting chat quality (Section 5). …

Content truncated to 500 words