DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9676–9691 July 9-14, 2023 ©2023 Association for Computational Linguistics DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering Pei Ke1, Fei Huang1, Fei Mi2, Yasheng Wang2, Qun Liu2, Xiaoyan Zhu1, Minlie Huang1∗ 1The CoAI Group, DCST, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2Huawei Noah’s Ark Lab, China kepei1106@outlook.com, f-huang18@mails.tsinghua.edu.cn {mifei2, wangyasheng, qun.liu}@huawei.com, {zxy-dcs, aihuang}@tsinghua.edu.cn Abstract Existing evaluation metrics for natural lan- guage generation (NLG) tasks face the chal- lenges on generalization ability and inter- pretability. Specifically, most of the well- performed metrics are required to train on eval- uation datasets of specific NLG tasks and evalu- ation dimensions, which may cause over-fitting to task-specific datasets. Furthermore, exist- ing metrics only provide an evaluation score for each dimension without revealing the evi- dence to interpret how this score is obtained. To deal with these challenges, we propose a simple yet effective metric called DecompE- val. This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained lan- guage models (PLMs) without training on eval- uation datasets, aiming to enhance the general- ization ability. To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that mea- sure the quality of each sentence. The subques- tions with their answers generated by PLMs are then recomposed as evidence to obtain the eval- uation result. Experimental results show that DecompEval achieves state-of-the-art perfor- mance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability1. 1 Introduction Recently, pre-trained language models (PLMs) such as GPT (Brown et al., 2020), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020) have achieved promising performance in natural lan- guage generation (NLG) tasks, such as text sum- marization (Zhang et al., 2020a) and dialogue gen- eration (Zhang et al., 2020c). As the quality of gen- ∗ Corresponding author 1The codes are available at https://github.com/ kepei1106/DecompEval erated texts gradually approaches that of human- written texts, there is an increasing demand for automatic evaluation metrics of generated texts. However, existing evaluation metrics are still struggling to measure the quality of generated texts accurately. Traditional metrics such as BLEU (Pap- ineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) rely on n-gram overlap between generated texts and reference texts, which fail to detect the issues in the content of gen- erated texts (Gehrmann et al., 2022). Recent works resort to model-based evaluation metrics to com- pute the similarity between generated texts and reference texts based on contextual representations from pre-trained models (Zhao et al., 2019; Zhang et al., 2020b) or adopt the score of language mod- eling (Yuan et al., 2021) / masked language mod- eling (Ke et al., 2022; Colombo et al., …

Content truncated to 500 words