Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619–3629 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3619 Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation Bo Pang1∗, Erik Nijkamp1∗, Wenjuan Han2∗§, Linqi Zhou1∗, Yixian Liu3, Kewei Tu3 1Department of Statistics, University of California, Los Angeles 2 School of Computing, National University of Singapore, Singapore 3School of Information Science and Technology, ShanghaiTech University {bopang, enijkamp, linqi.zhou}@ucla.edu dcshanw@nus.edu.sg {liuyx, tukw}@shanghaitech.edu.cn Abstract Open-domain dialogue generation has gained increasing attention in Natural Language Pro- cessing. Its evaluation requires a holistic means. Human ratings are deemed as the gold standard. As human evaluation is inefficient and costly, an automated substitute is highly desirable. In this paper, we propose holis- tic evaluation metrics that capture different as- pects of open-domain dialogues. Our metrics consist of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, (3) n-gram based di- versity in responses to augmented queries, and (4) textual-entailment-inference based logical self-consistency. The empirical validity of our metrics is demonstrated by strong correlations with human judgments. We open source the code and relevant materials. 1 1 Introduction Learning to communicate is a key capacity of in- telligent agents. Research on enabling a machine to have meaningful and natural conversations with humans plays a fundamental role in developing artificial general intelligence, as can be seen in the formulation of Turing test (Turing, 1950). Re- cently open-domain or non-task-oriented dialogue systems have attracted a surge of research interest (Bessho et al., 2012; Sordoni et al., 2015; Shang et al., 2015; Vinyals and Le, 2015; Serban et al., 2016, 2017; Ghazvininejad et al., 2018). Evaluating models of open-domain dialogue gen- eration in an efficient manner poses a significant challenge in developing dialogue systems. The prevalent method of open-domain dialogue eval- uation is human-based rating with a given rubric. ∗Equal contributions. §Wenjuan Han is the corresponding author. Wenjuan Han contributed to this work when at ShanghaiTech University. 1https://github.com/alexzhou907/ dialogue_evaluation. Context of Conversation Speaker A: Hey, what do you want to do tonight? Speaker B: Why don’t we go see a movie? Model Response Speaker A: Nah, let’s do something active. Reference Response Speaker A: Yeah, the film about Turing looks great! Context of Conversation Speaker A: Do you have maps of downtown area? Speaker B: Yes, here you are. Speaker A: How much is it? Model Response Speaker B: I’ve no idea, I don’t have any maps of the area. Table 1: Two responses from an dialogue system (Wolf et al., 2019) on Daily Dialogue Dataset. The first gen- erated response appears reasonable within the open- domain dialogue, while its BLEU score and semantic similarity between model response and reference re- sponse is low. The second generated response conflicts with its prior utterances. The italic text highlights the logical contradiction. When various variations in the model and sets of hyper-parameters are needed, the labor-intensive human evaluation is deemed impracticable. This key drawback may hinder the research …

Content truncated to 500 words