Planning with Learned Entity Prompts for Abstractive Summarization

Planning with Learned Entity Prompts for Abstractive Summarization Shashi Narayan Google Research shashinarayan@google.com Yao Zhao Google Brain yaozhaoyz@google.com Joshua Maynez Google Research joshuahm@google.com Gonc¸alo Sim˜oes Google Research gsimoes@google.com Vitaly Nikolaev Google Research vitalyn@google.com Ryan McDonald∗ ASAPP ryanmcd@asapp.com Abstract We introduce a simple but flexible mechanism to learn an intermediate plan to ground the gen- eration of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains—ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum, and BillSum, we demonstrate empirically that the grounded generation with the planning objec- tive improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of ROUGE. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to con- trol hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans. 1 Introduction Jones (1993) described text summarization—the task of generating accurate and concise summaries from source document(s)—as a three-step pro- cess: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation, and (iii) Synthesizing the output summary text. Com- mon to most traditional methods, an input rep- resentation was learned by semantically analyzing ∗Work done while Ryan McDonald was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was gen- erated grounded to the intermediate summary rep- resentation (Luhn, 1958; McKeown and Radev, 1995; Barzilay and Elhadad, 1997; Mihalcea and Tarau, 2004). State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architec- tures (seq2seq) with attention and copy mechanisms (Hochreiter and Schmidhuber, 1997; Bahdanau et al., 2015; See et al., 2017), Transformer archi- tectures with multi-headed self-attention (Vaswani et al., 2017), and large pretrained conditional lan- guage models (Dong et al., 2019; Song et al., 2019; Lewis et al., 2020; Rothe et al., 2020; Raffel et al., 2019; Zhang et al., 2020). However, the ground- ing of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2020; Raffel et al., 2019; Zhang et al., 2020), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in gener- ated summaries (Maynez et al., 2020; Kryscinski et al., 2020; Gabriel et al., 2021). In this paper, we investigate Entity Chains— ordered sequences of entities1 in the summary— as an intermediate summary representation to bet- ter plan and ground the generation of abstractive summaries. During training, we construct an aug- …

Content truncated to 500 words