PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment

Proceedings of the 29th International Conference on Computational Linguistics, pages 5896–5903 October 12–17, 2022. 5896 PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment Ge Luo and Hebi Li and Youbiao He and Forrest Sheng Bao Iowa State University Ames, IA, USA {gluo,hebi,yh54}@iastate.edu, forrest.bao@gmail.com Abstract Evaluating machine-generated summaries with- out a human-written reference summary has been a need for a long time. Inspired by pref- erence labeling in existing work of summariza- tion evaluation, we propose to judge summary quality by learning the preference rank of sum- maries using the Bradley-Terry power ranking model from inferior summaries generated by corrupting base summaries. Extensive experi- ments on several datasets show that our weakly supervised scheme can produce scores highly correlated with human ratings. 1 Introduction Summarization is a task in natural language pro- cessing in which automatic systems generate sum- maries from documents. To judge the quality of system-generated summaries, human evaluation is the best option, but it is non-trivial and labori- ous. Hence, many automatic metrics have been developed. They can be categorized as reference- based ones and reference-free ones, depending on whether reference summaries are needed in the evaluation stage. Reference-based metrics include ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), CIDEr (Vedantam et al., 2015), METEOR (Baner- jee and Lavie, 2005), S3 (Peyrard et al., 2017), MoverScore (Zhao et al., 2019), BertScore (Zhang et al., 2020), etc. Calculating the lexical over- lap or the embedding similarity between a system-generated summary and its corresponding human-written reference summary, they reportedly have high correlations with human assessments. Because creating human-written reference sum- maries is laborious and expensive, recent works are shifting to reference-free metrics. Sum- maQA (Scialom et al., 2019) and BLANC (Vasi- lyev et al., 2020) leverage pretrained language mod- els to carry out text understanding tasks to evaluate the helpfulness of a summary for understanding its source document. SUPERT (Gao et al., 2020b) measures the semantic similarity against a pseudo reference summary in a multi-document summa- rization setting. However, reference-free metrics may show a lower correlation (Fabbri et al., 2021) with human evaluation scores than some of the reference-based metrics. To trade off between the human effort needed and the quality of the evaluation, some work pur- sues a pairwise preference approach which collects preference labels over sentences in documents or over summaries from a human assessor as it re- quires less cognitive effort than writing a reference summary or manually scoring a machine-generated summary. Zopf (2018) proposes a reference-free evaluation approach by estimating sentence-level preferences on source documents rather than di- rectly on the generated summaries. Gao et al. (2020a) train a linear model to estimate a summary preference utility function via active preference learning to guide a reinforcement learning based summarization system. But they do not examine the learned preference model as a metric for sum- marization evaluation. Inspired by human-involved pairwise preference in summarization evaluation (Zopf, 2018; Gao et al., 2020b) and simple NLP data augmentation methods like EDA (Wei and Zou, 2019), in this work, …

Content truncated to 500 words