Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems

Proceedings of the 28th International Conference on Computational Linguistics, pages 4164–4178 Barcelona, Spain (Online), December 8-13, 2020 4164 Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems Vitou Phy♠, Yang Zhao♦, and Akiko Aizawa♥♠ ♠The University of Tokyo, Japan ♦IBM-Research Tokyo, Japan ♥National Institute of Informatics, Japan {vitou, aizawa}@nii.ac.jp, yangzhao@ibm.com Abstract Many automatic evaluation metrics have been proposed to score the overall quality of a response in open-domain dialogue. Generally, the overall quality is comprised of various aspects, such as relevancy, specificity, and empathy, and the importance of each aspect differs according to the task. For instance, specificity is mandatory in a food-ordering dialogue task, whereas fluency is preferred in a language-teaching dialogue system. However, existing metrics are not designed to cope with such flexibility. For example, BLEU score fundamentally relies only on word overlapping, whereas BERTScore relies on semantic similarity between reference and candidate response. Thus, they are not guaranteed to capture the required aspects, i.e., specificity. To design a metric that is flexible to a task, we first propose making these qualities manageable by grouping them into three groups: understandability, sensibleness, and likability, where likability is a combination of qualities that are essential for a task. We also propose a simple method to composite metrics of each aspect to obtain a single metric called USL-H, which stands for Understandability, Sensibleness, and Likability in Hierarchy1. We demonstrated that USL-H score achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics. 1 Introduction Evaluating a dialogue response is crucial for the development of open-domain dialogue systems. It allows comparison between different systems, which is similar to how the machine translation community uses BLEU (Papineni et al., 2002) to evaluate the overall quality of the translation and determines whether a system is the state-of-the-art (Bahdanau et al., 2015; Sennrich et al., 2016; Aharoni et al., 2019). Without automatic evaluation metrics, many studies tend to rely on either expert or crowdsourced platform to score the responses, which are both time-consuming and cost-ineffective (Zhang et al., 2018; Zhan et al., 2019; Adiwardana et al., 2020). To cope with this, various metrics have been proposed to score the overall quality of a dialogue response. Word overlap-based metrics, which were adopted from the MT community to measure the overlapping words between reference and candidate sentences, have been used to evaluate the dialogue responses (Sordoni et al., 2015; Zhang et al., 2018). However, Liu et al. (2016) showed that these metrics, i.e., BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), or ROUGE score (Lin, 2004), do not correlate well with human judgements, because there are many possible responses to reply to a given context. Recently, learning-based metrics, which aim to predict the overall quality of a response, have a better correlation with human judgment compared with word overlap-based metrics. Various training settings have also been explored. For example, ADEM (Lowe et al., 2017a) is trained to predict the score by learning to regress on human judgments, whereas …

Content truncated to 500 words