On Improving Summarization Factual Consistency from Natural Language Feedback

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15144–15161 July 9-14, 2023 ©2023 Association for Computational Linguistics On Improving Summarization Factual Consistency from Natural Language Feedback Yixin Liu∗1, Budhaditya Deb2, Milagro Teruel2, Aaron Halfaker2, Dragomir Radev1, Ahmed H. Awadallah2 1Yale University, 2Microsoft Research {yixin.liu, dragomir.radev}@yale.edu, {Budha.Deb, hassanam}@microsoft.com Abstract Despite the recent progress in language gen- eration models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural lan- guage can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual consistency in summa- rization, the quality that the summary should only contain information supported by the in- put documents, as the user-expected preference. We collect a high-quality dataset, DeFacto, containing human demonstrations and informa- tional natural language feedback consisting of corrective instructions, edited summaries, and explanations with respect to the factual consis- tency of the summary. Using our dataset, we study three natural language generation tasks: (1) editing a summary by following the human feedback, (2) generating human feedback for editing the original summary, and (3) revising the initial summary to correct factual errors by generating both the human feedback and edited summary. We show that DeFacto can provide factually consistent human-edited summaries and further insights into summarization factual consistency thanks to its informational natural language feedback. We further demonstrate that fine-tuned language models can leverage our dataset to improve the summary factual consistency, while large language models lack the zero-shot learning ability in our proposed tasks that require controllable text generation. 1 Introduction While recent natural language generation (NLG) models (Radford et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Brown et al., 2020) have made significant progress on the generation quality, they cannot always generate outputs that meet the user needs. For example, while state-of-the-art summa- rization systems can generate fluent and relevant ∗Most of the work was done while the first author was an intern at Microsoft Research. Figure 1: Three NLG tasks studied using our dataset. The Editing model aims to improve the initial system- generated summary given human feedback. The Critic model aims to predict human feedback according to a user-required quality. The Editor model aims to auto- matically correct factual errors by predicting both the human feedback and edited summary. summaries, recent work (Goyal and Durrett, 2021; Tang et al., 2022) have shown that they still make errors on fine-grained qualities such as factual con- sistency.1 These errors can lead to serious risks to the intended users and make it difficult for them to trust the systems for their decision-making. Such failures in satisfying the user needs have an intrinsic reason – the large benchmark datasets that are used to train NLG models are usually not collected according to pre-defined user needs, which results in a discrepancy between model behaviors and user expectations. For example, XSum (Narayan et al., 2018), one of the most commonly used summarization datasets, contains …

Content truncated to 500 words