To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3362–3376 August 1–6, 2021. ©2021 Association for Computational Linguistics 3362 To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text Matt Wilber Cornell University mw979@cornell.edu William Timkey Cornell University wpt25@cornell.edu Marten van Schijndel Cornell University mv443@cornell.edu Abstract Abstractive neural summarization models have seen great improvements in recent years, as shown by ROUGE scores of the gener- ated summaries. But despite these improved metrics, there is limited understanding of the strategies different models employ, and how those strategies relate their understand- ing of language. To understand this bet- ter, we run several experiments to character- ize how one popular abstractive model, the pointer-generator model of See et al. (2017), uses its explicit copy/generation switch to con- trol its level of abstraction (generation) vs ex- traction (copying). On an extractive-biased dataset, the model utilizes syntactic bound- aries to truncate sentences that are otherwise often copied verbatim. When we modify the copy/generation switch and force the model to generate, only simple paraphrasing abili- ties are revealed alongside factual inaccura- cies and hallucinations. On an abstractive- biased dataset, the model copies infrequently but shows similarly limited abstractive abili- ties. In line with previous research, these re- sults suggest that abstractive summarization models lack the semantic understanding neces- sary to generate paraphrases that are both ab- stractive and faithful to the source document. 1 Introduction Recent years have seen great improvements in “ab- stractive” summarization models – models that not only concatenate text from the source document, but can additionally paraphrase to generate sum- mary text. Once limited to sentence compression (Rush et al., 2015), abstractive models now gen- erate multi-sentence summaries (See et al., 2017), even for relatively long documents (Cohan et al., 2018). However, extractive models and mixed mod- els with significant extractive components continue to show strong performance, and the extent and manner in which abstraction is used by summariza- tion models is not well understood. Previous work has raised concerns about whether models are able to paraphrase in ways that lead to better summaries. Abstractive models often gen- erate summaries that are either ungrammatical or unfaithful to the source document (Maynez et al., 2020; Durmus et al., 2020; Kryscinski et al., 2020) and are prone to repetition in their outputs (See et al., 2019; Holtzman et al., 2020). These issues raise questions about how neural summarizers gen- erate novel text. Abstractive summarization is dif- ferentiated from extractive summarization by the model’s ability to paraphrase, but paraphrasing abil- ity is not directly measured by popular metrics, leading to a lack of understanding of the genera- tive process. Some previous research has aimed to alleviate these issues in evaluation: Zhang et al. (2018a) propose evaluating summaries with human evaluations of informativeness and coherence, and Ganesan (2018) implements a metric to reward models that paraphrase via simple synonym substi- tutions according to WordNet. However, synonym substitution is just one form of paraphrasing, and truly abstractive models should be capable of more complex …

Content truncated to 500 words