Assessing Dialogue Systems with Distribution Distances

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2192–2198 August 1–6, 2021. ©2021 Association for Computational Linguistics 2192 Assessing Dialogue Systems with Distribution Distances Jiannan Xiang1∗ Yahui Liu2∗ Deng Cai3 Huayang Li4 Defu Lian1 Lemao Liu4 1University of Science and Technology of China 2University of Trento, Italy 3The Chinese University of Hong Kong 4Tencent AI Lab, China Abstract An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing au- tomatic evaluation metrics are based on turn- level quality evaluation and use average scores for system-level comparison. In this paper, we propose to measure the performance of a di- alogue system by computing the distribution- wise distance between its generated conversa- tions and real-world conversations. Specifi- cally, two distribution-wise metrics, FBD and PRD, are developed and evaluated. Experi- ments on several dialogue corpora show that our proposed metrics correlate better with hu- man judgments than existing metrics. 1 Introduction Dialogue generation is a special text generation task, which has drawn booming attention in the nat- ural language processing community. It is widely agreed that one single input query is often asso- ciated with multiple valid responses in this task, which is termed as a 1-to-n relationship between a query and its responses (Vinyals and Le, 2015; Zhou et al., 2017; Zhao et al., 2017; Liu et al., 2018; Chen et al., 2021; Chan et al., 2021; Gao et al., 2021). It increases the challenges of auto- matically evaluating the performance of dialogue systems. In general, the previous evaluation metrics mainly focus on turn-level quality. For example, un- supervised word-overlapping or embedding-based metrics (Papineni et al., 2002; Lin, 2004; Mitchell and Lapata, 2008; Zhang et al., 2020) calculate the similarity or alignment between generated re- sponses and reference responses, which is not well- suited for open-end dialogue tasks. Learned classi- fication or regression systems (Lowe et al., 2017; ∗Equal contribution. Work was done during internship at Tencent AI Lab. Tao et al., 2018; Sellam et al., 2020; Ghazarian et al., 2019) are corpus-dependent because of re- quiring additional task-specific training or tuning, which run the risk of assigning lower quality to a better model in the overfitting or underfitting cases. In this paper, we provide a new perspective that distribution distance between generated conversa- tions and real conversations can be applied to mea- sure the performances of dialogue systems. There are three main contributions: (1) We firstly pro- pose two unsupervised distribution-wise metrics (i.e., FBD and PRD) to solve the evaluation is- sue in this field. (2) The experimental results show that the proposed distribution-wise metrics perform well. Particularly, FBD achieves compelling perfor- mances on most evaluation corpora, which shows a promising direction for designing evaluation met- rics. (3) We collect the typical evaluation corpora and existing evaluation metrics in order to better assess the performance of dialogue systems, which could be useful for researchers in this community 1. 2 Related Work In this section, we focus on unsupervised automatic evaluation …

Content truncated to 500 words