Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1363–1374 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1363 Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation Weixin Liang1, James Zou1 and Zhou Yu2 1 Stanford University,2 University of California, Davis {wxliang, jamesz}@stanford.edu, joyu@ucdavis.edu Abstract Open Domain dialog system evaluation is one of the most important challenges in dia- log research. Existing automatic evaluation metrics, such as BLEU are mostly reference- based. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Ama- zon Alexa Prize chatbots. However, self- reported user rating suffers from bias and vari- ance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an au- tomatic evaluation model CMADE (Compar- ison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user rat- ings as it trains on them. Specifically, we first use a self-supervised method to learn bet- ter dialog feature representation, and then use KNN and Shapley to remove confusing sam- ples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog com- parison task. Our implementation is available at https://github.com/Weixin-Liang/ dialog_evaluation_CMADE. 1 Introduction Open-domain dialog system evaluation is one of the most difficult challenges in the dialog commu- nity. Open-domain chatbots have a user-centric goal: to provide human with enjoyable user ex- perience. However, user experience is difficult to quantify due to bias and variance among different users. Previous research has optimized on auto- matic dialog evaluation metrics such as BLUE (Pa- pineni et al., 2002), which measures the difference between the generated responses and the reference responses. Due to the contrast between the one-to- many nature of open-domain conversations and the limited number of available references, such met- rics correlate poorly with human judgments (Liu et al., 2016; Lowe et al., 2017; Novikova et al., 2017). Designing a fully automatic dialog evalua- tion metric is still an open research problem. Currently, both academia and industry (Ram et al., 2018a; Li et al., 2019b; Liang et al., 2019) rely on human ratings to evaluate open-domain dialogs. Following the ubiquitous application of Likert scores in survey research like online re- views (Godes and Silva, 2012) and consumer sat- isfaction (Peterson and Wilson, 1992), a common practice of human evaluation on dialogs is to ask ei- ther a third-person rater or the chatbot user to report a Likert score. However, concerns have been raised about the validity of Likert score-based ratings. Ku- likov et al. (Kulikov et al., 2018) observe high bias and variance of Likert scores. Such issue is more se- vere in real-world commercial dialog systems like Alexa social chatbot (Ram et al., 2018a; Venkatesh et al., 2018), because the real-world users have nei- ther monetary incentive nor necessary annotation training to calibrate their ratings. To explore the validity of Likert score based dialog …

Content truncated to 500 words