FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3336 - 3355 December 7-11, 2022 ©2022 Association for Computational Linguistics FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation Chen Zhang†,⋆ Luis Fernando D’Haro‡ Qiquan Zhang† Thomas Friedrichs⋆ Haizhou Li♥,†,♢ †National University of Singapore ⋆Robert Bosch (SEA), Singapore ‡Universidad Politécnica de Madrid, Spain ♢Kriston AI Lab, China ♥The Chinese University of Hong Kong, Shenzhen, China chen_zhang@u.nus.edu Abstract Recent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment1. However, they either perform turn-level evalua- tion or look at a single dialogue quality dimen- sion. One would expect a good evaluation met- ric to assess multiple quality dimensions at the dialogue level. To this end, we are motivated to propose a multi-dimensional dialogue-level metric, which consists of three sub-metrics with each targeting a specific dimension. The sub- metrics are trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimen- sions. Moreover, we explore two approaches to combine the sub-metrics: metric ensemble and multitask learning. Both approaches yield a holistic metric that significantly outperforms individual sub-metrics. Compared to the exist- ing state-of-the-art metric, the combined met- rics achieve around 16% relative improvement on average across three high-quality dialogue- level evaluation benchmarks. 1 Introduction In the study of generative dialogue systems, we heavily rely on some reference-based static metrics, such as BLEU (Papineni et al., 2002), to measure improvements during system development and to compare various model variants. These metrics still need improvement due to their poor correlations with human judgment (Liu et al., 2016) and poor interpretability (Mehri and Eskenazi, 2020b). Recently, model-based reference-free met- rics (Yeh et al., 2021) represent one of the ways to address the limitations of static reference-based metrics. Although such metrics exhibit promis- ing correlations with human evaluation, most of 1As shown in (Yeh et al., 2021), most reference-free met- rics can achieve around 0.3 to 0.6 Spearman correlations on various turn-level benchmarks. However, on the dialogue-level benchmarks, most metrics perform poorly (< 0.2 Spearman correlations). them (Tao et al., 2018; Ghazarian et al., 2019; Huang et al., 2020; Sinha et al., 2020; Mehri and Es- kenazi, 2020b; Phy et al., 2020; Pang et al., 2020; Zhang et al., 2021c) target turn-level evaluation, i.e., they focus on single-response quality, such as contextual relevance and naturalness. When evaluating a multi-turn human-chatbot dialogue, turn-level metrics do not model the dialogue in to- tality, but frame it as a set of context-response pairs. They assign scores to every chatbot responses in the dialogue. Hence, an aggregation strategy is required to derive the single dialogue-level metric score, such as taking the average of all the response- level scores. Both prior works (Zhang et al., 2021a; Yeh et al., 2021) and our experimental results in §5 suggest that such an approach yields sub-optimal dialogue-level evaluation. The reason may be that turn-level metrics do not model the dependency among utterances within multi-turn interactions, it is difficult for them to spot errors that are only …

Content truncated to 500 words