Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6009–6017 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6009 Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization Ramya Akula, Ivan Garibay University of Central Florida Orlando, Florida, U.S.A ramya.akula@knights.ucf.edu, igaribay@ucf.edu Abstract The development of an automatic evaluation metric remains an open problem in text generation. Widely used evaluation metrics, like ROUGE and BLEU, are based on exact word matching and fail to capture semantic similarity. Recent works, such as BERTScore, MoverScore and, Sentence Mover’s Similarity, are an improvement over these standard metrics as they use contextualized word or sentence embeddings to capture semantic similarity. We in this work, propose a novel evaluation metric, Sentence Pair EmbEDdings (SPEED) Score, for text generation which is based on semantic similarity between sentence pairs as opposed to earlier approaches. To find semantic similarity between a pair of sentences, we obtain sentence-level embeddings from multiple transformer models pre-trained specifically on various sentence pair tasks such as Paraphrase Detection (PD), Semantic Text Similarity (STS), and Natural Language Inference (NLI). As these sentence pair tasks involve capturing the semantic similarity between a pair of input texts, we leverage these models in our metric computation. Our proposed evaluation metric shows impressive performance in evaluating both abstractive and extractive summarization models and achieves state-of-the-art results on the SummEval dataset, demonstrating the effectiveness of our approach. Also, we perform the run-time analysis to show that our proposed metric is faster than the current state-of-the-art. Keywords: Evaluation Metric, Abstractive Summarization, Extractive Summarization, Semantic Similarity 1. Introduction Developing an efficient automatic evaluation metrics for text summarization will have a two-fold impact on the advancement of algorithms for text generation sys- tems. First, it eliminates the need for time-consuming and expensive human evaluations, second, it helps for a better comparison and assessment of the developed systems. With this goal, various evaluation metrics have been proposed in the literature to automatically evaluate summarization methods. Widely used metrics: ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002), rely on word/token overlap between reference and sys- tem summary. Recent works such as BERTScore (Zhang et al., 2019), MoverScore (Zhao et al., 2019) are improvements over earlier methods as they use word/token embeddings from pre-trained language models instead of using ex- act word/token matches. However, a drawback of these metrics is that they do not capture the perfor- mance of the models in terms of semantic consistency of the generated summary to its reference (Clark et al., 2019). This shortcoming becomes more prominent as the length of the summaries increase. To overcome these issues and to better evaluate summarization mod- els, there is a need for a metric that can find semantic similarity between the ground truth and system sum- mary at a level beyond the individual words/tokens. In this work, we propose the idea of using sentence- pair embeddings to compare multi-sentence summaries for automatic evaluation of text summarization mod- els. To extract the …

Content truncated to 500 words