Exploring Content Selection in Summarization of Novel Chapters

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5043–5054 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5043 Exploring Content Selection in Summarization of Novel Chapters Faisal Ladhak1∗, Bryan Li2∗, Yaser Al-Onaizan3, Kathleen McKeown1,3 1Columbia University, 2University of Pennsylvania, 3Amazon AI faisal.ladhak@columbia.edu, bryanli@seas.upenn.edu, onaizan@amazon.com, kathy@cs.columbia.edu Abstract We present a new summarization task, gener- ating summaries of novel chapters using sum- mary/chapter pairs from online study guides. This is a harder task than the news summa- rization task, given the chapter length as well as the extreme paraphrasing and generaliza- tion found in the summaries. We focus on extractive summarization, which requires the creation of a gold-standard set of extractive summaries. We present a new metric for align- ing reference summary sentences with chap- ter sentences to create gold extracts and also experiment with different alignment methods. Our experiments demonstrate significant im- provement over prior alignment approaches for our task as shown through automatic met- rics and a crowd-sourced pyramid analysis. 1 Introduction When picking up a novel one is reading, it would be helpful to be reminded of what happened last. To address this need, we develop an approach to generate extractive summaries of novel chap- ters. This is much harder than the news summa- rization tasks on which most of the summariza- tion field (e.g., (Cheng and Lapata, 2016; Grusky et al., 2018; Paulus et al., 2017)) focuses; chap- ters are on average seven times longer than news articles. There is no one-to-one correspondence between summary and chapter sentences, and the summaries in our dataset use extensive paraphras- ing, while news summaries copy most of their in- formation from the words used in the article. We focus on the task of content selection, tak- ing an initial, extractive summarization approach given the task difficulty.1 As the reference sum- ∗Equal contribution. Work done while at Amazon. 1We tried two abstractive models (Chen and Bansal, 2018; Liu and Lapata, 2019) but ROUGE was low and the output was poor with many repetitions and hallucinations. maries are abstractive, training our model re- quires creating a gold-standard set of extractive summaries. We present a new approach for aligning chapter sentences with the abstractive summary sentences, incorporating weighting to ROUGE (Lin, 2004) and METEOR (Lavie and Denkowski, 2009) metrics to enable the alignment of salient words between them. We also experi- ment with BERT (Devlin et al., 2018) alignment. We use a stable matching algorithm to select the best alignments, and show that enforcing one-to- one alignments between reference summary sen- tences and chapter sentences is the best alignment method of those used in earlier work. We obtain a dataset of summaries from five study guide websites paired with chapter text from Project Gutenberg. Our dataset consists of 4,383 unique chapters, each of which is paired with two to five human-written summaries. We experiment with generating summaries us- ing our new alignment method within three mod- els that have been developed for single document news summarization (Chen …

Content truncated to 500 words