Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7419–7451 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7419 Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach Haoming Jiang∗ 1,2 , Bo Dai3, Mengjiao Yang3, Tuo Zhao2, Wei Wei4 1Amazon.com Inc, CA, USA 3Google Brain, CA, USA2Georgia Institute of Technology, GA, USA 4Google Cloud AI, CA, USA {jianghm,tourzhao}@gatech.edu {bodai,sherryy,wewei}@google.com Abstract Reliable automatic evaluation of dialogue sys- tems under an interactive environment has long been overdue. An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve human interac- tion, which is usually not affordable for large scale experiments. Though researchers have attempted to use metrics for language gener- ation tasks (e.g., perplexity, BLEU) or some model-based reinforcement learning methods (e.g., self-play evaluation) for automatic eval- uation, these methods only show very weak correlation with the actual human evaluation in practice. To bridge such a gap, we propose a new framework named ENIGMA for estimat- ing human evaluation scores based on recent advances of off-policy evaluation in reinforce- ment learning. ENIGMA only requires a hand- ful of pre-collected experience data, and there- fore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More impor- tantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experi- ence data (see details in Section 2), which sig- nificantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors. Our experiments show that ENIGMA significantly outperforms exist- ing methods in terms of correlation with hu- man evaluation scores. 1 Introduction One of the fundamental research bottlenecks for de- veloping dialog systems falls in evaluation, namely how to measure the performance of these systems in an automatic and scalable manner. Different from supervised natural language understanding tasks (e.g., text classification and machine transla- tion), an ideal environment for evaluating dialog systems, also known as the Turing test, involves multi-turn human interaction (Turing, 1950; Liu ∗ Work was done during internship at Google Cloud AI. et al., 2016; Ghandeharioun et al., 2019; See et al., 2019). While online platforms such as Amazon Mechanical Turk can provide human-based evalua- tion, they are often expensive and not scalable. Researchers have adopted language quality met- rics for single-turn response generation given a fixed context (e.g., BLEU score and perplexity) to implement automatic dialog systems evalua- tion (DeVault et al., 2011; Xiang et al., 2014; Hi- gashinaka et al., 2014; Gandhe and Traum, 2016; Lowe et al., 2017). However, these metrics only weakly correlate to human evaluation in prac- tice (Liu et al., 2016; Ghandeharioun et al., 2019). One cause of such weak correlation is that language quality metrics rely on the exact match between generated text and ground-truth, which generally do not fully overlap. While certain embedding- based metrics have been developed to combat this lack of coverage (Mitchell and Lapata, 2008; Dziri et al., 2019), they are only post-hoc judgments based on static experience data, …

Content truncated to 500 words