TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1865–1874 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1865 TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising Ziyi Yang1⇤, Chenguang Zhu2⇤, Robert Gmyr2, Michael Zeng2, Xuedong Huang2, Eric Darve1 Stanford University1 {zy99,darve}@stanford.edu Microsoft Cognitive Services Research Group2 {chezhu,rogmyr,nzeng,xdh}@microsoft.com Abstract Text summarization aims to extract essential information from a piece of text and trans- form the text into a concise version. Existing unsupervised abstractive summarization mod- els leverage recurrent neural networks frame- work while the recently proposed transformer exhibits much more capability. Moreover, most of previous summarization models ig- nore abundant unlabeled corpora resources available for pretraining. In order to address these issues, we propose TED, a transformer- based unsupervised abstractive summariza- tion system with pretraining on large-scale data. We first leverage the lead bias in news articles to pretrain the model on millions of un- labeled corpora. Next, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the qual- ity of generated summaries. Notably, TED outperforms all unsupervised abstractive base- lines on NYT, CNN/DM and English Giga- word datasets with various document styles. Further analysis shows that the summaries gen- erated by TED are highly abstractive, and each component in the objective function of TED is highly effective. 1 Introduction Summarization refers to the task of condensing a document into a shorter version without losing the key information. Summarization models can be categorized into two types: abstractive and extrac- tive. Extractive models select sentences from the input article as the summary. Such process ensures a basic level of grammaticality and accuracy, but also limits the model ability to copying. In con- trast, abstractive models summarize a document using newly generated tokens and phrases that may not be found in the original article, which involves ⇤Equal contribution. Work was done during first author’s internship at Microsoft. a process requiring an advanced ability to refine, paraphrase and re-organize language information (See et al., 2017; Narayan et al., 2018; Gunel et al., 2020). Like most machine learning algorithms, sum- marization models can also be divided into super- vised and unsupervised categories. Supervised ap- proaches require in-domain parallel data, i.e. both input articles and corresponding reference sum- maries must be present for the teacher-forcing train- ing (Hermann et al., 2015; Liu and Lapata, 2019). Unfortunately, high-quality paired data are not al- ways available across different text domains and styles. Moreover, considering the fact that summa- rization is not an easy task even for people, reli- able human-labeled data are also difficult to obtain. Therefore, several unsupervised summarization ap- proaches have been proposed, which do not require reference summaries for the target domain. We introduce these methods as follows. Unsupervised extractive models. TextRank (Mihalcea and Tarau, 2004) encodes sentences in the article as nodes in an undirected graph. The weights of edges are measured by sentences simi- larity. The centrality of a node (sentence) is com- puted by PageRank (Brin and Page, 1998) …

Content truncated to 500 words