SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1347–1354 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1347 SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization Yang Gao†, Wei Zhao‡, Steffen Eger‡ † Dept. of Computer Science, Royal Holloway, University of London, UK ‡ Computer Science Department, Technische Universit¨at Darmstadt, Germany yang.gao@rhul.ac.uk, {zhao,eger}@aiphes.tu-darmstadt.de Abstract We study unsupervised multi-document sum- marization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, rat- ings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo refer- ence summary, i.e. selected salient sentences from the source documents, using contextu- alized embeddings and soft token alignment techniques. Compared to the state-of-the- art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18- 39%. Furthermore, we use SUPERT as re- wards to guide a neural-based reinforcement learning summarizer, yielding favorable per- formance compared to the state-of-the-art un- supervised summarizers. All source code is available at https://github.com/yg211/ acl20-ref-free-eval. 1 Introduction Evaluating the quality of machine-generated sum- maries is a highly laborious and hence expensive task. Most existing evaluation methods require certain forms of human involvement, thus are su- pervised: they either directly let humans rate the generated summaries (e.g. Pyramid (Nenkova and Passonneau, 2004)), elicit human-written reference summaries and measure their overlap with the gen- erated summaries (e.g. using ROGUE (Lin, 2004a) or MoverScore (Zhao et al., 2019)), or collect some human annotations (e.g. preferences over pairs of summaries (Gao et al., 2019a)) to learn a sum- mary evaluation function. Evaluation in multi- document summarization is particularly expensive: Lin (2004b) reports that it requires 3,000 hours of human effort to evaluate the summaries from the Document Understanding Conferences (DUC)1. 1http://duc.nist.gov/ Salient Sentences Extractor Semantic Similarity Measurement Summary relevance score Pseudo reference summary Summary to evaluate Doc 1 Doc 2 Doc N Figure 1: Workflow of SUPERT. To reduce the expenses for evaluating multi- document summaries, we investigate unsupervised evaluation methods, which require neither human annotations nor reference summaries. In particu- lar, we focus on evaluating the relevance (Peyrard, 2019) of multi-document summaries, i.e. measur- ing how much salient information from the source documents is covered by the summaries. There ex- ist a few unsupervised evaluation methods (Louis and Nenkova, 2013; Sun and Nenkova, 2019), but they have low correlation with human relevance rat- ings at summary level: given multiple summaries for the same source documents, these methods can hardly distinguish summaries with high relevance from those with low relevance (see §3). Contributions. First, to better measure the se- mantic overlap between source documents and machine-generated summaries, we propose to use state-of-the-art contextualized text encoders, e.g. BERT (Devlin et al., 2019) and its variant Sentence- BERT (SBERT) (Reimers and Gurevych, 2019), which is optimized for measuring semantic simi- larity between sentences, to develop unsupervised evaluation methods. We measure the relevance of a summary in two steps: (i) identifying the salient in- formation in the input …

Content truncated to 500 words