HOLMS: Alternative Summary Evaluation with Large Language Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 5679–5688 Barcelona, Spain (Online), December 8-13, 2020 5679 HOLMS: Alternative Summary Evaluation with Large Language Models Yassine Mrabet mrabety@mail.nih.gov Dina Demner-Fushman ddemner@mail.nih.gov U.S. National Library of Medicine National Institutes of Health 8600 Rockville Pike, Bethesda, MD, 20894 Abstract Efficient document summarization requires evaluation measures that can not only rank a set of systems based on an average score, but also highlight which individual summary is better than another. However, despite the very active research on summarization approaches, few works have proposed new evaluation measures in the recent years. The standard measures relied upon for the development of summarization systems are most often ROUGE and BLEU which, despite being efficient in overall system ranking, remain lexical in nature and have a limited potential when it comes to training neural networks. In this paper, we present a new hybrid evaluation measure for summarization, called HOLMS, that combines both language models pre-trained on large corpora and lexical similarity measures. Through several experiments, we show that HOLMS outperforms ROUGE and BLEU substantially in its correlation with human judgments on several extractive summarization datasets for both linguistic quality and pyramid scores. 1 Introduction . Generating human readable summaries of textual documents is posed to remain a key technology in our information era. Whether summarizing news, scientific articles, encyclopedias, or social media posts, the demand for a faster consumption of the most relevant information is expected to grow hand in hand with the amount of information available online. A current bottleneck to a wider adoption of automatic summarizers is the lack of efficient solutions addressing both the relevance of the generated summaries and their linguistic quality. One component of this current limitation is the lack of efficient evaluation measures that address both aspects. The two most cited and widely adopted measures in various summarization datasets and summarization challenges are ROUGE (Lin and Hovy, 2003; Lin, 2004) and BLEU (Papineni et al., 2002). While both measures have been shown to be highly efficient baselines in ranking summarization systems based on their average score (Dang and Vanderwende, 2007; Dang and Owczarzak, 2008; Dang and Owczarzak, 2009; Owczarzakg, 2010; Owczarzak and Dang, 2011), fewer studies have examined their relevance for individual summary ranking and their adequacy with linguistic quality and fluency. With the new levels of performance achieved by neural language models in a variety of natural language processing tasks, several insights point towards the high potential of their contextual language encoding for language representation. Most of the state-of-the-art models such as T5 (Raffel et al., 2019), BERT (Devlin et al., 2019), GPT (Radford et al., 2018; Radford et al., 2019) and the Universal Sentence Encoder (Cer et al., 2018), are built from very large corpora, which reduces substantially the potential bias from using them to evaluate summaries in a restricted document set or benchmark. On the other hand, the shallow lexical features of the original texts play a key role in extractive summarization and in pointer-generator approaches. It …

Content truncated to 500 words