Simple Unsupervised Summarization by Contextual Matching

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5101–5106 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5101 Simple Unsupervised Summarization by Contextual Matching Jiawei Zhou Harvard University jzhou02@g.harvard.edu Alexander M. Rush Harvard University srush@seas.harvard.edu Abstract We propose an unsupervised method for sen- tence summarization using only language modeling. The approach employs two lan- guage models, one that is generic (i.e. pre- trained), and the other that is specific to the tar- get domain. We show that by using a product- of-experts criteria these are enough for main- taining continuous contextual matching while maintaining output fluency. Experiments on both abstractive and extractive sentence sum- marization data sets show promising results of our method without being exposed to any paired data. 1 Introduction Automatic text summarization is the process of formulating a shorter output text than the origi- nal while capturing its core meaning. We study the problem of unsupervised sentence summa- rization with no paired examples. While data- driven approaches have achieved great success based on various powerful learning frameworks such as sequence-to-sequence models with atten- tion (Rush et al., 2015; Chopra et al., 2016; Nalla- pati et al., 2016), variational auto-encoders (Miao and Blunsom, 2016), and reinforcement learning (Paulus et al., 2017), they usually require a large amount of parallel data for supervision to do well. In comparison, the unsupervised approach reduces the human effort for collecting and annotating large amount of paired training data. Recently researchers have begun to study the unsupervised sentence summarization tasks. These methods all use parameterized unsuper- vised learning methods to induce a latent vari- able model: for example Schumann (2018) uses a length controlled variational autoencoder, Fevry and Phang (2018) use a denoising autoencoder but only for extractive summarization, and Wang and Lee (2018) apply a reinforcement learning pro- cedure combined with GANs, which takes a fur- ther step to the goal of Miao and Blunsom (2016) using language as latent representations for semi- supervised learning. This work instead proposes a simple approach to this task that does not require any joint training. We utilize a generic pretrained language model to enforce contextual matching between sentence prefixes. We then use a smoothed problem spe- cific target language model to guide the fluency of the generation process. We combine these two models in a product-of-experts objective. This ap- proach does not require any task-specific training, yet experiments show results on par with or better than the best unsupervised systems while produc- ing qualitatively fluent outputs. The key aspect of this technique is the use of a pretrained language model for unsupervised contextual matching, i.e. unsupervised paraphrasing. 2 Model Description Intuitively, a sentence summary is a shorter sen- tence that covers the main point succinctly. should satisfy the following two properties (simi-It lar to Pitler (2010)): (a) Faithfulness: the sequence is close to the original sentence in terms of mean- ing; (b) Fluency: the sequence is grammatical and sensible to the domain. We propose to …

Content truncated to 500 words