How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation Chia-Wei Liu1∗, Ryan Lowe1∗, Iulian V. Serban2∗, Michael Noseworthy1∗, Laurent Charlin1, Joelle Pineau1 1 School of Computer Science, McGill University {chia-wei.liu,ryan.lowe,michael.noseworthy}@mail.mcgill.ca {lcharlin, jpineau}@cs.mcgill.ca 2 DIRO, Universit´e de Montr´eal iulian.vlad.serban@umontreal.ca Abstract We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not avail- able. Recent works in response generation have adopted metrics from machine transla- tion to compare a model’s generated response to a single target response. We show that these metrics correlate very weakly with hu- man judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain. We provide quantitative and quali- tative results highlighting specific weaknesses in existing metrics, and provide recommenda- tions for future development of better auto- matic evaluation metrics for dialogue systems. 1 Introduction An important aspect of dialogue response generation systems, which are trained to produce a reasonable utterance given a conversational context, is how to evaluate the quality of the generated response. Typi- cally, evaluation is done using human-generated su- pervised signals, such as a task completion test or a user satisfaction score (Walker et al., 1997; M¨oller et al., 2006; Kamm, 1995), which are relevant when the dialogue is task-focused. We call models opti- mized for such supervised objectives supervised di- alogue models, while those that do not are unsuper- vised dialogue models. This paper focuses on unsupervised dialogue re- sponse generation models, such as chatbots. These ∗Denotes equal contribution. models are receiving increased attention, partic- ularly using end-to-end training with neural net- works (Serban et al., 2016; Sordoni et al., 2015; Vinyals and Le, 2015). This avoids the need to col- lect supervised labels on a large scale, which can be prohibitively expensive. However, automatically evaluating the quality of these models remains an open question. Automatic evaluation metrics would help accelerate the deployment of unsupervised re- sponse generation systems. Faced with similar challenges, other natural lan- guage tasks have successfully developed automatic evaluation metrics. For example, BLEU (Papineni et al., 2002a) and METEOR (Banerjee and Lavie, 2005) are now standard for evaluating machine translation models, and ROUGE (Lin, 2004) is often used for automatic summarization. These metrics have recently been adopted by dialogue researchers (Ritter et al., 2011; Sordoni et al., 2015; Li et al., 2015; Galley et al., 2015b; Wen et al., 2015; Li et al., 2016). However these metrics assume that valid responses have significant word overlap with the ground truth responses. This is a strong assump- tion for dialogue systems, where there is significant diversity in the space of valid responses to a given context. This is illustrated in Table 1, where two rea- sonable responses are proposed to the context, but these responses do not share any words in common and do not have …

Content truncated to 500 words