REAM♯: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2487–2500 August 1–6, 2021. ©2021 Association for Computational Linguistics 2487 REAM♯: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation Jun Gao Tencent AI Lab, Shenzhen, China imgaojun@gmail.com Wei Bi∗ Tencent AI Lab, Shenzhen, China victoriabi@tencent.com Ruifeng Xu∗ Peng Cheng Laboratory, Shenzhen, China xuruifeng.hitsz@gmail.com Shuming Shi Tencent AI Lab, Shenzhen, China shumingshi@tencent.com Abstract The lack of reliable automatic evaluation met- rics is a major impediment to the development of open-domain dialogue systems. Various reference-based metrics have been proposed to calculate a score between a predicted re- sponse and a small set of references. How- ever, these metrics show unsatisfactory correla- tions with human judgments. For a reference- based metric, its reliability mainly depends on two factors: its ability to measure the simi- larity between the predicted response and the reference response, as well as the reliability of the given reference set. Yet, there are few discussions on the latter. Our work attempts to fill this vacancy. We first clarify an as- sumption on reference-based metrics that, if more high-quality references are added into the reference set, the reliability of the met- ric will increase. Next, we present REAM♯: an enhancement approach to Reference-based EvAluation Metrics1 for open-domain dia- logue systems. A prediction model is designed to estimate the reliability of the given refer- ence set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric. Experiments validate both the effectiveness of our prediction model and that the reliability of reference-based metrics improves with the augmented reference sets. 1 Introduction The lack of reliable automatic evaluation metrics is a major impediment to the development of open- domain dialogue systems (Li and Jurafsky, 2016; Gao et al., 2019; Li et al., 2020a). The under- lying difficulty in evaluation lies in the diversity of the possible outcomings. Existing evaluation metrics for open-domain dialogue systems can be ∗Corresponding authors 1Interested reader may contact the authors to obtain a copy of the code and the data. roughly divided into reference-based and reference- free metrics. Reference-based metrics usually mea- sure how similar a generated response is to the reference responses. Reference-free metrics, on the other hand, measure the quality of a response without any reference and usually focus on specific aspects of the responses. For example, much work often computes the perplexity of a generated re- sponse as a measure of fluency (Li et al., 2020b), and adopts Dist-1/2 (Li et al., 2016b) to measure the diversity of the response. In this work, we focus on reference-based metrics. BLEU (Papineni et al., 2002), originally for machine translation, is now a popular reference- based metric to evaluate open-domain dialog sys- tems automatically. However, it has been shown that BLEU and other word-overlap metrics such as METEOR (Banerjee and Lavie, 2005) and ROUGE (Lin, 2004), rely on surface-form simi- larities only without considering the semantic di- versity, thus fail to correlate well with human …

Content truncated to 500 words