Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8640–8665 July 9-14, 2023 ©2023 Association for Computational Linguistics Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method Yiming Wang, Zhuosheng Zhang, Rui Wang∗ Shanghai Jiao Tong University alsaceym@gmail.com, {zhangzs, wangrui12}@sjtu.edu.cn Abstract Automatic summarization generates concise summaries that contain key ideas of source documents. As the most mainstream datasets for the news sub-domain, CNN/DailyMail and BBC XSum have been widely used for per- formance benchmarking. However, the refer- ence summaries of those datasets turn out to be noisy, mainly in terms of factual hallucina- tion and information redundancy. To address this challenge, we first annotate new expert- writing Element-aware test sets following the “Lasswell Communication Model” proposed by Lasswell (1948), allowing reference sum- maries to focus on more fine-grained news el- ements objectively and comprehensively. Uti- lizing the new test sets, we observe the surpris- ing zero-shot summary ability of LLMs, which addresses the issue of the inconsistent results between human preference and automatic eval- uation metrics of LLMs’ zero-shot summaries in prior work. Further, we propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step, which helps them integrate more fine- grained details of source documents into the final summaries that correlate with the human writing mindset. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two datasets, respectively. Dataset and code are publicly available at https://github.com/Alsace08/SumCoT. 1 Introduction Automatic summarization is a challenging text gen- eration task that condenses the source text into a few coherent and abstract sentences. In recent years, the study of summarization has evolved with supervised learning based on sequence-to-sequence architectures (Sutskever et al., 2014; Vinyals et al., 2015; Vaswani et al., 2017) and transfer learn- ing based on pre-trained language models (De- ∗ Corresponding author On March 8, Marcin Wasniewski crashed into the back of a lorry on the A444 in Coventry when driving a car. Incredibly, he was injured with just cuts and bruises and cheated death by just millimeters. Paramedics were shocked because of serious damage to the car. This father firmly believed that Jesus saved him. Element-aware Summary (Ours) Entity Date Event Result Trailer 'embedded' into car windscreen in smash on A444 in Coventry. Impact would have 'certainly been fatal' if a couple of inches closer to driver. Paramedics were shocked when Marcin Wasniewski walked out unaided. Dataset-specific Summary (Original) Figure 1: Case comparisons for our Element-aware summary and original dataset-specific summary. News elements have been highlighted with different color shadows. It is clear that our element-aware summary covers more comprehensive elements, and the logical connection between the elements is smoother. vlin et al., 2019; Zhang et al., 2019; Liu et al., 2019; Lewis et al., 2020). Existing studies com- monly train or fine-tune language models on large- scale corpus (Nallapati et al., 2016; Narayan et al., 2018; Koupaee and Wang, 2018; Fabbri et al., 2019), so …

Content truncated to 500 words