GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9230–9240, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9230 GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems Lishan Huang1∗, Zheng Ye1*, Jinghui Qin1, Liang Lin1,2, Xiaodan Liang1,2† 1 Sun Yat-Sen University, 2 Dark Matter AI Inc. {huanglsh6,yezh7,qinjingh}@mail2.sysu.edu.cn, linliang@ieee.org, xdliang328@gmail.com Abstract Automatically evaluating dialogue coherence is a challenging but high-demand ability for de- veloping high-quality open-domain dialogue systems. However, current evaluation met- rics consider only surface features or utterance- level semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. Here, we first consider that the graph structure constituted with topics in a dialogue can accurately depict the underlying communication logic, which is a more natural way to produce persuasive metrics. Capital- ized on the topic-level dialogue graph, we pro- pose a new evaluation metric GRADE, which stands for Graph-enhanced Representations for Automatic Dialogue Evaluation. Specif- ically, GRADE incorporates both coarse- grained utterance-level contextualized repre- sentations and fine-grained topic-level graph representations to evaluate dialogue coherence. The graph representations are obtained by rea- soning over topic-level dialogue graphs en- hanced with the evidence from a common- sense graph, including k-hop neighboring rep- resentations and hop-attention weights. Exper- imental results show that our GRADE signif- icantly outperforms other state-of-the-art met- rics on measuring diverse dialogue models in terms of the Pearson and Spearman correla- tions with human judgements. Besides, we release a new large-scale human evaluation benchmark to facilitate future research on au- tomatic metrics. 1 Introduction Coherence, what makes dialogue utterances uni- fied rather than a random group of sentences, is an essential property to pursue an open-domain ∗Equal Contribution. †Corresponding Author. Why not use the treadmill? Or maybe get a dog? Sometimes my husband goes with me. I like the outdoors. So , do you enjoy eating too ? My love of eating is why I exercise. Score treadmill dog outdoors eat exercise evidences context response GRADE outdoors exercise treadmill dog eat active fat sport campers hiking Commonsense Graph Dialogue Graph Figure 1: An illustrative example of how our GRADE evaluates dialogue coherence by incorporating graph information on topic transitions from a commonsense graph. Topic keywords of the context and the response are highlighted in green and red respectively, which can be aligned to the corresponding nodes in the com- monsense graph. The white nodes and all the edges in the commonsense graph are pieces of evidence that assist in constructing the dialogue graph. Taking ad- vantage of such evidence, GRADE can better capture the topic transition dynamics between the context and the response, as shown in the thickness of edges in the dialogue graph. dialogue system aiming at conversing with hu- mans. Although open-domain dialogue systems have achieved significant progress and performed much more human-like skills in recent years (Zhou et al., 2020; Adiwardana et al., 2020; Roller et al., 2020), automatically measuring dialogue coher- ence for state-of-the-art open-domain dialogue models is still an open and under-explored research problem attributing to the …

Content truncated to 500 words