DynaEval: Unifying Turn and Dialogue Level Evaluation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5676–5689 August 1–6, 2021. ©2021 Association for Computational Linguistics 5676 DynaEval: Unifying Turn and Dialogue Level Evaluation Chen Zhang†,⋆ Yiming Chen† Luis Fernando D’Haro‡ Yan Zhang† Thomas Friedrichs⋆ Grandee Lee† Haizhou Li†,⋆⋆ †National University of Singapore ⋆Robert Bosch (SEA), Singapore ‡Universidad Polit´ecnica de Madrid, Spain ⋆⋆Kriston AI Lab, China {chen zhang,yiming.chen,grandee.lee}@u.nus.edu, {haizhou.li,eleyanz}@nus.edu.sg, luisfernando.dharo@upm.es, thomas.friedrichs@sg.bosch.com Abstract A dialogue is essentially a multi-turn interac- tion among interlocutors. Effective evaluation metrics should reflect the dynamics of such interaction. Existing automatic metrics are focused very much on the turn-level quality, while ignoring such dynamics. To this end, we propose DynaEval1, a unified automatic eval- uation framework which is not only capable of performing turn-level evaluation, but also holistically considers the quality of the entire dialogue. In DynaEval, the graph convolu- tional network (GCN) is adopted to model a dialogue in totality, where the graph nodes de- note each individual utterance and the edges represent the dependency between pairs of ut- terances. A contrastive loss is then applied to distinguish well-formed dialogues from care- fully constructed negative samples. Experi- ments show that DynaEval significantly out- performs the state-of-the-art dialogue coher- ence model, and correlates strongly with hu- man judgements across multiple dialogue eval- uation aspects at both turn and dialogue level. 1 Introduction Modern dialogue systems (Smith et al., 2020; Zhang et al., 2020; Adiwardana et al., 2020) lever- aging large-scale language model pre-training (De- vlin et al., 2019; Radford et al., 2019) are capable of generating fluent and contextually relevant utter- ances. Yet, they still face difficulties in mimicking human conversations in the sense that they lack certain conversation-level attributes, such as coher- ence (Cervone et al., 2018), consistency (Welleck et al., 2019; Nie et al., 2020), diversity (Li et al., 2016; Wu et al., 2020) and engagement (Ghande- harioun et al., 2019; Ghazarian et al., 2020). One of the main reasons is the dearth of effective dialogue- level evaluation mechanisms to guide the studies and to monitor progress. 1https://github.com/e0397123/DynaEval Commonly used static metrics, such as BLEU (Papineni et al., 2002), ME- TEOR (Denkowski and Lavie, 2014) and ROUGE (Lin, 2004), correlate poorly with human judgements (Liu et al., 2016) rendering them unsuitable for dialogue evaluation. While some recent automatic dialogue evaluation met- rics (Ghazarian et al., 2019; Mehri and Eskenazi, 2020b; Huang et al., 2020; Zhang et al., 2021b) demonstrate strong correlations with human judgement at the turn-level, they only focus on context-response pairs without explicitly modeling the interaction over an entire dialogue. To perform dialogue-level evaluation, we need to rely on the aggregation of turn-level scores over the dialogue as a proxy for a dialogue-level score. Furthermore, a recent study by Mehri and Eske- nazi (2020a) found out that even though state-of- the-art chatbots outperform humans across multiple turn-level evaluation criteria, such as interesting- ness, engagement and specificity, their dialogue- level ratings like coherence, Likability and diver- sity are …

Content truncated to 500 words