Towards a Unified Multi-Dimensional Evaluator for Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023 - 2038 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards a Unified Multi-Dimensional Evaluator for Text Generation Ming Zhong§ Yang Liu† Da Yin♣ Yuning Mao§ Yizhu Jiao§ Pengfei Liu‡ Chenguang Zhu† Heng Ji§ Jiawei Han§ §University of Illinois at Urbana-Champaign ♣University of California, Los Angeles†Microsoft Cognitive Services Research ‡Carnegie Mellon University {mingz5, yuningm2, yizhuj2, hengji, hanj}@illinois.edu {yaliu10, chezhu}@microsoft.com da.yin@cs.ucla.edu pliu3@cs.cmu.edu Abstract Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Lan- guage Generation (NLG), i.e., evaluating the generated text from multiple explainable di- mensions, such as coherence and fluency. How- ever, automatic evaluation in NLG is still dom- inated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional eval- uator UNIEVAL for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with dif- ferent questions, we can use one evaluator to evaluate from multiple dimensions. Further- more, thanks to the unified Boolean QA format, we are able to introduce an intermediate learn- ing phase that enables UNIEVAL to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UNIEVAL correlates substantially better with human judg- ments than existing metrics. Specifically, com- pared to the top-performing unified evaluators, UNIEVAL achieves a 23% higher correlation on text summarization, and over 43% on di- alogue response generation. Also, UNIEVAL demonstrates a strong zero-shot learning abil- ity for unseen evaluation dimensions and tasks. Source code, data and all pre-trained evaluators are available on our GitHub repository1. 1 Introduction The rapid development of Natural Language Gen- eration (NLG) tasks with the support of pre-trained language models (Raffel et al., 2020; Brown et al., 2020; Lewis et al., 2020) calls for a higher qual- ity evaluation of generated texts. However, the evaluation process is still dominated by traditional similarity-based metrics (Kasai et al., 2021), ex- emplified by ROUGE (Lin, 2004) and BLEU (Pa- pineni et al., 2002) that compute n-gram overlap 1https://github.com/maszhongming/UniEval between the model output and the reference text. These metrics are potentially misleading as NLG models have advanced to the point where discrep- ancies between them are unlikely to be detected based on surface-level features (Gehrmann et al., 2022). Although using pre-trained models to obtain embedding-based similarity may alleviate this is- sue (Zhang et al., 2019), these metrics still naturally lead to the question: does similarity to reference text indicate the overall quality of model output? Belz and Gatt (2008) referred to this similarity as “human-likeness” and pointed out that the ability to output human-like text may be completely unre- lated to the final performance on generation tasks. Realizing that creating a one-size-fits-all score is infeasible, subsequent research has focused on a more comprehensive multi-dimensional evalua- tion for NLG tasks. It aims to evaluate the model output from multiple explainable dimensions and has been …

Content truncated to 500 words