Focus Attention: Promoting Faithfulness and Diversity in Summarization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6078–6095 August 1–6, 2021. ©2021 Association for Computational Linguistics 6078 Focus Attention: Promoting Faithfulness and Diversity in Summarization Rahul Aralikatte∗ University of Copenhagen rahul@di.ku.dk Shashi Narayan Google Research shashinarayan@google.com Joshua Maynez Google Research joshuahm@google.com Sascha Rothe Google Research rothe@google.com Ryan McDonald∗ ASAPP ryanmcd@asapp.com Abstract Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultane- ously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage de- coders to proactively generate tokens that are similar or topical to the input document. Fur- ther, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summa- rization task, two state-of-the-art models aug- mented with Focus Attention generate sum- maries that are closer to the target and more faithful to their input documents, outperform- ing their vanilla counterparts on ROUGE and multiple faithfulness measures. We also em- pirically demonstrate that Focus Sampling is more effective in generating diverse and faith- ful summaries than top-k or nucleus sampling- based decoding methods. 1 Introduction Document summarization — producing the shorter version of a document while preserving salient in- formation (Mani, 2001; Nenkova and McKeown, 2011) — is challenging even for humans. Today, systems can generate summaries with a high level of fluency and coherence. This is due to recent advances such as sequence-to-sequence architec- tures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997; Bahdanau et al., 2015; Gu et al., 2016), fully attention-based Transformer architectures (Vaswani et al., 2017), and large pretrained language models (Devlin et al., ∗Work done when authors were interning/working at Google. A GOLD: Australia has expelled an Israeli diplomat saying Israel was behind the forging of Australian passports linked to the murder of a Hamas operative in Dubai. PEGASUS: Australia has expelled an Israeli diplomat after concluding that forged Australian passports used in the killing of a Hamas militant in Dubai were issued by Israel. Our PEGFAME model: The Australian government has expelled an Israeli diplomat over the use of forged Australian passports in the killing of a Hamas militant in Dubai. B PEGASUS with Top-k Sampling Israel has summoned the Australian ambassador to complain after the Australian government said forged passports used in the killing of a Hamas operative in Dubai belonged to Netanyahu’s foreign ministry. The Australian government has ordered Israel to withdraw an officer over the use of forged Australian passports used by the 2013 murder of a Lebanese opposition figure in Dubai. PEGASUS with Nucleus Sampling Israel hasracuse withdrawn an envoy after the Australian government said it concluded that Israeli agents used forged passports used to kill a Dubai …

Content truncated to 500 words