Enhancing the Open-Domain Dialogue Evaluation in Latent Space

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4889–4900 August 1–6, 2021. ©2021 Association for Computational Linguistics 4889 Enhancing the Open-Domain Dialogue Evaluation in Latent Space Zhangming Chan1,3,∗, Lemao Liu2, Juntao Li4, Haisong Zhang2, Dongyan Zhao1,3, Shuming Shi2 and Rui Yan5 1 Wangxuan Institute of Computer Technology, Peking University 2 Tencent AI Lab 3 Center for Data Science, AAIS, Peking University 4 School of Computer Science and Technology, Soochow University 5 Gaoling School of Artificial Intelligence, Renmin University of China {zhangming.chan,zhaody}@pku.edu.cn ljt@suda.edu.cn, ruiyan@ruc.edu.cn {redmondliu,hansonzhang,shumingshi}@tencent.com Abstract The notorious one-to-many nature of open- domain dialogues poses huge challenges for automatic evaluation methods. Recent stud- ies attempt to mitigate this issue by consid- ering the similarity of the generated response with the conversational context and design dis- criminative models to learn from multiple pos- itive responses. Despite the promising results, they can not be applied to general scenarios where training data with multiple responses is unavailable. To this end, in this paper, we propose a self-supervised setting to obtain a smooth latent space that can both capture discourse-level context information and im- plicitly model more references in latent space. Specifically, we present EMS, an Enhanced di- alogue evaluation Metric in latent Space. Ex- perimental results on two real-world dialogue datasets confirm the superiority of our method for open-domain dialogue evaluation, where both Pearson and Spearman correlations with human judgments outperform all baselines. 1 Introduction With the surge of deep learning techniques, generation-based open-domain dialogue systems have witnessed significant improvement in re- cent years. Plenty of novel and effective mod- els (Sutskever et al., 2014; Serban et al., 2016; Li et al., 2015; Serban et al., 2016; Zhao et al., 2017; Gu et al., 2018; Qiu et al., 2019; Chan et al., 2019b; Serban et al., 2017; Wolf et al., 2019; Hu et al., 2019; Chen et al., 2020) are proposed and have greatly promoted the development of the open- domain dialogue generation. Unlike the endless emergence of novel methods, however, there is still no meaningful and widely accepted automatic eval- uation metric for dialogue generation yet. As we ∗ This work was done while Z. Chan was an intern at Tencent AI Lab. Corresponding Author: Rui Yan. know, automatic evaluation allows quick and ef- fective comparison between different systems and is crucial for the development of natural language generation (NLG) tasks (Dathathri et al., 2019; Gu et al., 2019; Gao et al., 2019; Chan et al., 2019a, 2020). The lack of meaningful automatic evalua- tion metrics has become a significant impediment for open-domain dialog generation research. Over the past decade, many automatic evalu- ation metrics are proposed to evaluate the open- domain dialogue systems. Among them, the word overlap-based automatic evaluation metrics from NLG tasks, such as BLEU (Papineni et al., 2002) in machine translation and ROUGE (Lin, 2004) in text summarization, are popular. In addition, Embedding Metrics (Mitchell and Lapata, 2008; Forgues et al., 2014; Rus and Lintean, 2012) have been utilized to evaluate the open-domain dialogue systems (Gu et …

Content truncated to 500 words