Learning an Unreferenced Metric for Online Dialogue Evaluation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2430–2441 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2430 Learning an Unreferenced Metric for Online Dialogue Evaluation Koustuv Sinha∗, 1,2,3 Prasanna Parthasarathi, 1,2 Jasmine Wang, 1 Ryan Lowe, 1,2,4 William L. Hamilton, 1,2 and Joelle Pineau 1,2,3 1 School of Computer Science, McGill University, Canada 2 Quebec Artificial Intelligence Institute (Mila), Canada 3 Facebook AI Research (FAIR), Montreal, Canada 4 OpenAI Abstract Evaluating the quality of a dialogue interaction between two agents is a difficult task, espe- cially in open-domain chit-chat style dialogue. There have been recent efforts to develop au- tomatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference re- sponse during inference, making it infeasible for online evaluation. Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to ex- tract latent representations of utterances, and leverages the temporal transitions that exist be- tween them. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true re- sponses for comparison during inference. 1 Introduction Recent approaches in deep neural language genera- tion have opened new possibilities in dialogue gen- eration (Serban et al., 2017; Weston et al., 2018). Most of the current language generation efforts are centered around language modelling or machine translation (Ott et al., 2018), which are evaluated by comparing directly against the reference sen- tences. In dialogue, however, comparing with a single reference response is difficult, as there can be many reasonable responses given a context that have nothing to do with each other (Liu et al., 2016). Still, dialogue research papers tend to report scores based on word-overlap metrics from the machine translation literature (e.g. BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014)). However word-overlap metrics aggressively penal- ize the generated response based on lexical differ- ences with the ground truth and correlate poorly to human judgements (Liu et al., 2016). ∗Corresponding author: koustuv.sinha@mail.mcgill.ca. Code for reproducing the experiments are available at https://github.com/facebookresearch/online dialog eval. Figure 1: Model architecture for MaUdE, which is an unsupervised unreferenced metric for dialog evalu- ation. One can build dialogue evaluation metrics in two ways: referenced metrics, which compare the generated response with a provided ground-truth re- sponse (such as the above word-overlap metrics), or an unreferenced metrics, which evaluate the gener- ated response without any such comparison. Lowe et al. (2017) propose a learned referenced metric named ADEM, which learns an alignment score be- tween context and response to predict human score annotations. However, since the score is trained to mimic human judgements, it requires collect- ing large-scale human annotations on the dataset in question and cannot be easily applicable to new datasets (Lowe, 2019). Recently, Tao et al. (2017) proposed a hybrid referenced-unreferenced metric named RUBER, where the metric is trained without requiring hu- man responses by bootstrapping negative samples directly from the dataset. However, referenced met- …

Content truncated to 500 words