Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1784–1795 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1784 Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers Shusheng Xu1∗, Xingxing Zhang2, Yi Wu1,3, Furu Wei2 and Ming Zhou2 1 IIIS, Tsinghua Unveristy, Beijing, China 2 Microsoft Research Asia, Beijing, China 3 Shanghai Qi Zhi institute, Shanghai China xuss20@mails.tsinghua.edu.cn {xizhang,fuwei,mingzhou}@microsoft.com jxwuyi@gmail.com Abstract Unsupervised extractive document summariza- tion aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. In this work, we find that transformer attentions can be used to rank sentences for unsuper- vised extractive summarization. Specifically, we first pre-train a hierarchical transformer model using unlabeled documents only. Then we propose a method to rank sentences using sentence-level self-attentions and pre-training objectives. Experiments on CNN/DailyMail and New York Times datasets show our model achieves state-of-the-art performance on unsu- pervised summarization. We also find in ex- periments that our model is less dependent on sentence positions. When using a linear combi- nation of our model and a recent unsupervised model explicitly modeling sentence positions, we obtain even better results. 1 Introduction Document summarization is the task of transform- ing a long document into its shorter version while still retaining its important content. Researchers have explored many paradigms for summarization, while the most popular ones are extractive summa- rization and abstractive summarization (Nenkova and McKeown, 2011). As their names suggest, ex- tractive summarization generates summiries by ex- tracting text from original documents, and abstrac- tive summarization rewrites documents by para- phrasing or deleting some words or phrases. Most summarization models require labeled data, where documents are paired with human written summaries. Unfortunately, human labeling for sum- marization task is expensive and therefore high ∗ Work done during the first author’s internship at Microsoft Research Asia. quality large scale labeled summarization datasets are rear (Hermann et al., 2015) compared to grow- ing web documents created everyday. It is also not possible to create summaries for documents in all text domains and styles. In this paper, we focus on unsupervised summarization, where we only need unlabeled documents during training. Many attempts for unsupervised summariza- tion are extractive (Carbonell and Goldstein, 1998; Radev et al., 2000; Lin and Hovy, 2002; Mihal- cea and Tarau, 2004; Erkan and Radev, 2004; Wan, 2008; Wan and Yang, 2008; Hirao et al., 2013; Parveen et al., 2015). The core problem is to iden- tify salient sentences in a document. The most popular approaches among these work rank sen- tences in the document using graph based algo- rithms, where each node is a sentence and weights of edges are measured by sentence similarities. Then a graph ranking method is employed to es- timate sentence importance. For example, Tex- tRank (Mihalcea and Tarau, 2004) utilizes word co-occurrence statistics to compute similarity and then employs PageRank (Page et al., 1997) to rank sentences. Sentence similarities in (Zheng and La- pata, 2019) …

Content truncated to 500 words