Towards Quantifiable Dialogue Coherence Evaluation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2718–2729 August 1–6, 2021. ©2021 Association for Computational Linguistics 2718 Towards Quantifiable Dialogue Coherence Evaluation Zheng Ye1, Liucun Lu1, Lishan Huang2, Liang Lin2,3, Xiaodan Liang1∗ 1Shenzhen Campus of Sun Yat-sen University, 2Sun Yat-Sen University, 3Dark Matter AI Inc. {yezh7,lulc,huanglsh6}@mail2.sysu.edu.cn, linliang@ieee.org, xdliang328@gmail.com Abstract Automatic dialogue coherence evaluation has attracted increasing attention and is crucial for developing promising dialogue systems. How- ever, existing metrics have two major limita- tions: (a) they are mostly trained in a simpli- fied two-level setting (coherent vs. incoherent), while humans give Likert-type multi-level co- herence scores, dubbed as “quantifiable”; (b) their predicted coherence scores cannot align with the actual human rating standards due to the absence of human guidance during train- ing. To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating stan- dards. Specifically, QuantiDCE includes two training stages, Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) fine-tuning. During MLR pre-training, a new MLR loss is proposed for enabling the model to learn the coarse judgement of coherence degrees. Then, during KD fine-tuning, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated data. To advo- cate the generalizability even with limited fine- tuning data, a novel KD regularization is intro- duced to retain the knowledge learned at the pre-training stage. Experimental results show that the model trained by QuantiDCE presents stronger correlations with human judgements than the other state-of-the-art metrics. 1 1 Introduction Dialogue coherence, which requires a response to be fluent, consistent and context-related, is an es- sential property for developing promising dialogue ∗Corresponding Author. 1The code and trained checkpoints are available at https: //github.com/James-Yip/QuantiDCE. 1 3 2 5 4 Context Response U1: I like animals, especially dogs. How about you? U2: Haha, I like cats more. I like cats more too and I work in a pet store. U3: Do your work for a pet store? Very coherent? Mostly coherent? Mostly incoherent? Very incoherent? …… 0 1 Coherent? Incoherent? …… Figure 1: Likert-type multi-level human rating vs. two- level automatic evaluation. Human rating always con- siders multiple coherence degrees, while most of the existing automatic metrics only learn to distinguish the coherence dialogues from the incoherent ones and give relatively extreme coherence scores. systems (Cervone et al., 2018). However, it is still challenging to evaluate the coherence of a response generated by a dialogue system. Although human evaluation is always considered as the most accu- rate way to evaluate the coherence, it is expensive and high-latency, which cannot meet the evaluation demand of the frequent development of dialogue systems. Therefore, automatic evaluation metrics are developed to serve as human proxies that can rapidly compute the dialogue coherence and return relatively accurate results. The current widely used metrics measure the lexical word-overlap between generated …

Content truncated to 500 words