Stepwise Extractive Summarization and Planning with Structured Transformers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4143–4159, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4143 Stepwise Extractive Summarization and Planning with Structured Transformers Shashi Narayan∗ Joshua Maynez∗ Jakub Adamek {shashinarayan,joshuahm,enkait}@google.com Daniele Pighin Bla˘z Bratani˘c Ryan McDonald {biondo,blazb,ryanmcd}@google.com Google Research Abstract We propose encoder-centric stepwise mod- els for extractive summarization using struc- tured transformers – HiBERT (Zhang et al., 2019) and Extended Transformers (Ainslie et al., 2020). We enable stepwise summa- rization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure. Our models are not only efficient in modeling the structure of long inputs, but they also do not rely on task-specific redundancy-aware modeling, making them a general purpose extractive con- tent planner for different tasks. When evalu- ated on CNN/DailyMail extractive summariza- tion, stepwise models achieve state-of-the-art performance in terms of Rouge without any redundancy aware modeling or sentence filter- ing. This also holds true for Rotowire table- to-text generation, where our models surpass previously reported metrics for content selec- tion, planning and ordering, highlighting the strength of stepwise modeling. Amongst the two structured transformers we test, stepwise Extended Transformers provides the best per- formance across both datasets and sets a new standard for these challenges.1 1 Introduction Extractive document summarization is the task of creating a summary by identifying (and sub- sequently concatenating) the most important sen- tences in a document (Erkan and Radev, 2004; Nenkova and McKeown, 2011). In recent years this task has matured significantly, mostly thanks to advances in deep neural networks. Cheng and Lapata (2016) conceptualize extractive summariza- tion as a sequence labeling task in which first a hier- archical long short-term memory network (LSTM; ∗ Equal contribution. 1The code and data are available at https://github. com/google-research/google-research/ tree/master/etcsum. Hochreiter and Schmidhuber, 1997) is used to en- code a document and then another LSTM is used to predict for each sentence whether it should be included in the summary. This architecture was later adopted by Nallapati et al. (2016a), Nallapati et al. (2017), Narayan et al. (2018b), Zhang et al. (2018) and Dong et al. (2018). Following the success of pre-trained transformer- based architectures for many tasks (Vaswani et al., 2017; Devlin et al., 2019), the current state-of-the- art approach to extractive summarization uses trans- formers to learn sentence representations and to rank sentences by their saliency (Liu, 2019; Liu and Lapata, 2019b; Zhang et al., 2019; Zhong et al., 2019a; Bi et al., 2020). The top scoring sentences are then assembled to produce an extract of the doc- ument. Summaries built in this fashion (Cheng and Lapata, 2016; Narayan et al., 2018a; Zhang et al., 2018; Dong et al., 2018) are prone to contain redun- dant information. Several recent approaches have explored mechanisms to better handle redundancy, such as heuristic-based Trigram Blocking (TriBlk; Liu and Lapata, 2019b; Wang et al., 2020), hand- crafted feature-driven models (Ren et al., 2017) and redundancy aware neural sequence models (Zhou et al., 2018; Bi et al., …

Content truncated to 500 words