Evaluating Coherence in Dialogue Systems using Entailment

Proceedings of NAACL-HLT 2019, pages 3806–3812 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3806 Evaluating Coherence in Dialogue Systems using Entailment Nouha Dziri∗ Ehsan Kamalloo∗ Kory W. Mathewson Osmar Zaiane Department of Computing Science University of Alberta {dziri,kamalloo,korym,zaiane}@cs.ualberta.ca Abstract Evaluating open-domain dialogue systems is difficult due to the diversity of possible correct answers. Automatic metrics such as BLEU correlate weakly with human annotations, re- sulting in a significant bias across different models and datasets. Some researchers re- sort to human judgment experimentation for assessing response quality, which is expensive, time consuming, and not scalable. Moreover, judges tend to evaluate a small number of dia- logues, meaning that minor differences in eval- uation configuration may lead to dissimilar re- sults. In this paper, we present interpretable metrics for evaluating topic coherence by mak- ing use of distributed sentence representations. Furthermore, we introduce calculable approx- imations of human judgment based on conver- sational coherence by adopting state-of-the-art entailment techniques. Results show that our metrics can be used as a surrogate for human judgment, making it easy to evaluate dialogue systems on large-scale datasets and allowing an unbiased estimate for the quality of the re- sponses. 1 Introduction Recently, we have witnessed a big success in the capability of computers to seemingly understand natural language text and to generate plausible re- sponses to conversations (Serban et al., 2016; Xing et al., 2017; Sordoni et al., 2015; Li et al., 2016; Serban et al., 2017; Devlin et al., 2018; Radford et al., 2018). A challenging task of building dia- logue systems lies in evaluating the quality of their responses. Typically, evaluating goal-oriented di- alogue systems is done via human-generated judg- ment like a task completion test or user satisfaction score (Walker et al., 1997; M¨oller et al., 2006). However, the task of evaluating open-ended dia- logue systems is not well defined as there is no ∗Equal Contribution clear explicit goal for conversations. Indeed, di- alog systems are ultimately created to satisfy the user’s need which can be associated with how en- tertaining and engaging the conversation was. It is unclear how to define a metric that can ac- count comprehensibly for the semantic meaning of the responses. Moreover, grasping the underly- ing meaning of text has always been fraught with difficulties, which are essentially attributed to the complexities and ambiguities in natural language. Generally, a good dialogue can be described as an exchange of information that sustain coherence through a train of thoughts and a flow of topics. Therefore, a plausible way to evaluate open-ended dialogue systems is to measure the consistency of the responses. For example, a neural dialogue system can respond to the utterance “Do you like animals?” by “Yes, I have three cats”, thereafter replies to “How many cats do you have” by “I don’t have cats.”. Here, we can notice that the di- alogue system failed to provide a coherent answer and instead generated an inconsistent response. In this work, we characterize the consistency …

Content truncated to 500 words