Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1116–1126 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1103 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1116–1126 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1103 Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses Ryan Lowe♥∗ Michael Noseworthy♥∗ Iulian V. Serban♦ Nicolas A.-Gontier♥ Yoshua Bengio♦‡ Joelle Pineau♥‡ ♥ Reasoning and Learning Lab, School of Computer Science, McGill University ♦ Montreal Institute for Learning Algorithms, Universit´e de Montr´eal ‡ CIFAR Senior Fellow Abstract Automatically evaluating the quality of di- alogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with hu- man judgements of response quality. Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evalua- tion as a learning problem. We present an evaluation model (ADEM) that learns to pre- dict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model’s predictions correlate significantly, and at a level much higher than word-overlap met- rics such as BLEU, with human judge- ments at both the utterance and system- level. We also show that ADEM can gener- alize to evaluating dialogue models unseen during training, an important step for auto- matic dialogue evaluation. 1 Introduction Building systems that can naturally and meaning- fully converse with humans has been a central goal of artificial intelligence since the formulation of the Turing test (Turing, 1950). Research on one type of such systems, sometimes referred to as non-task-oriented dialogue systems, goes back to the mid-60s with Weizenbaum’s famous program ELIZA: a rule-based system mimicking a Roge- rian psychotherapist by persistently either rephras- ing statements or asking questions (Weizenbaum, ∗ Indicates equal contribution. Context of Conversation Speaker A: Hey, what do you want to do tonight? Speaker B: Why don’t we go see a movie? Model Response Nah, let’s do something active. Reference Response Yeah, the film about Turing looks great! Figure 1: Example where word-overlap scores fail for dialogue evaluation; although the model response is reasonable, it has no words in common with the reference response, and thus would be given low scores by metrics such as BLEU. 1966). Recently, there has been a surge of inter- est towards building large-scale non-task-oriented dialogue systems using neural networks (Sordoni et al., 2015b; Shang et al., 2015; Vinyals and Le, 2015; Serban et al., 2016a; Li et al., 2015). These models are trained in an end-to-end manner to op- timize a single objective, usually the likelihood of generating the responses from a fixed corpus. Such models have already had a substantial im- pact in industry, including Google’s Smart Reply system (Kannan et al., 2016), and Microsoft’s Xi- aoice chatbot …

Content truncated to 500 words