RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12856–12875 July 9-14, 2023 ©2023 Association for Computational Linguistics RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue Zhengliang Shi1, Weiwei Sun1, Shuo Zhang2, Zhen Zhang1, Pengjie Ren1, Zhaochun Ren1∗ 1Shandong University, Qingdao, China 2Bloomberg, London, United Kingdom shizhl@mail.sdu.edu.cn {sunnweiwei, zhen.zhang.sdu}@gmail.com zhaochun.ren@sdu.edu.cn szhang611@bloomberg.net jay.ren@outlook.com Abstract Evaluating open-domain dialogue systems is challenging for reasons such as the one-to- many problem, i.e., many appropriate re- sponses other than just the golden response. As of now, automatic evaluation methods need bet- ter consistency with humans, while reliable hu- man evaluation can be time- and cost-intensive. To this end, we propose the Reference-Assisted Dialogue Evaluation (RADE) approach un- der the multi-task learning framework, which leverages the pre-created utterance as reference other than the gold response to relief the one-to- many problem. Specifically, RADE explicitly compares reference and the candidate response to predict their overall scores. Moreover, an auxiliary response generation task enhances prediction via a shared encoder. To support RADE, we extend three datasets with addi- tional rated responses other than just a golden response by human annotation. Experiments on our three datasets and two existing benchmarks demonstrate the effectiveness of our method, where Pearson, Spearman, and Kendall correla- tions with human evaluation outperform state- of-the-art baselines. 1 Introduction Open-domain dialogue system, which focuses on non-goal-oriented chitchat, may converse on a broad range of arbitrary topics. Recent years have witnessed rapid advances in natural language gener- ation (Zhang et al., 2019b; Roller et al., 2021; Zhao et al., 2023), boosting the development of open- domain dialogue systems. Conversations with such systems resemble human-human interactions as various responses might fit the context, given that users often do not have a specific goal beyond enjoying the conversation. Evaluating these con- versations is thus challenging because of the so- called one-to-many problem (Chan et al., 2021; Ji et al., 2022); see Figure 1 where three candidate ∗∗ Corresponding author. What's your favorite online store? I like to shop online, probably a bit too much. I love to shop on Amazon. where do you shop? I shop at Amazon and Barnes & Noble. Reference: I shop online at Amazon. It's great. I shop online a lot too. Amazon is good but expensive. I prefer Barnes & Noble. 1 2 3 1 > 2 3 > Human 3 > 1 2 > BLEU 3 > 2 1 > BERTScore Figure 1: An example to explain the one-to-many nature of open-domain dialogues. responses with different semantics fit the context while there is only one golden response. The most common practice of dialogue evalua- tion is done with reference-based metrics, which compare the generated response with a pre-created response, commonly referred to as the golden standard (Ji et al., 2022). The reference-based metrics calculate the similarity between the gen- erated and gold responses at either lexical level (e.g., ROUGE (Lin, 2004), BLEU (Papineni et al., 2002)) or semantic level (e.g., BERTScore (Zhang et al., 2019a), ADEM …

Content truncated to 500 words