Toward Human-Like Evaluation for Natural Language Generation with Error Analysis

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5892–5907 July 9-14, 2023 ©2023 Association for Computational Linguistics Toward Human-Like Evaluation for Natural Language Generation with Error Analysis Qingyu Lu1,2∗, Liang Ding2∗, Liping Xie1, Kanjian Zhang1†, Derek F. Wong3, Dacheng Tao4 1School of Automation, Southeast University 2JD Explore Academy 3NLP2CT Lab, University of Macau 4The University of Sydney {luqingyu,lpxie,kjzhang}@seu.edu.cn, liangding.liam@gmail.com, derekfw@um.edu.mo, dacheng.tao@gmail.com Abstract The pretrained language model (PLM) based metrics have been successfully used in evaluating language generation tasks. Recent studies of the human evaluation community show that considering both major errors (e.g. mistranslated tokens) and minor errors (e.g. imperfections in fluency) can produce high-quality judgments. This inspires us to approach the final goal of the automatic metrics (human-like evaluations) by fine-grained error analysis. In this paper, we argue that the ability to estimate sentence confidence is the tip of the iceberg for PLM-based metrics. And it can be used to refine the generated sentence toward higher confidence and more reference-grounded, where the costs of refining and approaching reference are used to deter- mine the major and minor errors, respectively. To this end, we take BARTScore as the testbed and present an innovative solution to marry the unexploited sentence refining capacity of BARTScore and human-like error analysis, where the final score consists of both the evaluations of major and minor errors. Experiments show that our solution consis- tently improves BARTScore, outperforming top-scoring metrics in 19/25 test settings. Analyses demonstrate our method robustly and efficiently approaches human-like evaluations, enjoying better interpretability. Our code and scripts will be publicly released in https: //github.com/Coldmist-Lu/ ErrorAnalysis_NLGEvaluation. 1 Introduction Leveraging the power of large pre-trained language models (PLMs) has been proven effective in eval- uating natural language generation (NLG) tasks (Ma et al., 2019; Mathur et al., 2020b). Metrics like BERTScore (Zhang et al., 2020b) and Mover- ∗Work was done when Qingyu was interning at JD Ex- plore Academy. Qingyu and Liang contributed equally. †Corresponding Author. Source: 迈克去书店。 Reference: Mike goes to the bookstore. Hypothesis: Jerry goes to bookstore happily. Iteration Refined Sentence BARTScore (↑) 0 Jerry -15.16 goes -1.64 to -0.48 bookstore -5.30 happily -14.51 . -0.02 -3.89 1 Mike -4.06 goes -1.56 to -0.54 bookstore -5.50 happily -14.42 . -0.03 -2.59 2 Mike -4.06 goes -1.56 to -0.54 bookstore -5.50 . -0.06 -1.45 Table 1: An example of error analysis framework, specifically, detect-correct algorithm in §3.2. Scores under each token represent the log probability assigned by BARTScore. Worse tokens detected by error analysis in each iteration are highlighted in yellow, and their corresponding scores are in red. score (Zhao et al., 2019) leverage contextual em- beddings provided by PLMs to evaluate the seman- tic similarity of sentences. Regression-based met- rics like COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020) introduce a regression layer following PLMs to learn a supervised prediction using human evaluation. Recently, another line of research focuses on generation probabilities of seq2seq PLMs to measure the confidence of gen- erated …

Content truncated to 500 words