BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3752–3761, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3752 BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle Peter West1 Ari Holtzman1,2 Jan Buys1 Yejin Choi1,2 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Allen Institute for Artificial Intelligence {pawest, ahai, jbuys, yejin}@cs.washington.edu Abstract The principle of the Information Bottleneck (Tishby et al., 1999) is to produce a summary of information X optimized to predict some other relevant information Y . In this paper, we propose a novel approach to unsupervised sen- tence summarization by mapping the Informa- tion Bottleneck principle to a conditional lan- guage modelling objective: given a sentence, our approach seeks a compressed sentence that can best predict the next sentence. Our iter- ative algorithm under the Information Bottle- neck objective searches gradually shorter sub- sequences of the given sentence while max- imizing the probability of the next sentence conditioned on the summary. Using only pre- trained language models with no direct super- vision, our approach can efficiently perform extractive sentence summarization over a large corpus. Building on our unsupervised extractive sum- marization (BottleSumEx), we then present a new approach to self-supervised abstrac- tive summarization (BottleSumSelf), where a transformer-based language model is trained on the output summaries of our unsupervised method. Empirical results demonstrate that our extractive method outperforms other un- supervised models on multiple automatic met- rics. In addition, we find that our self- supervised abstractive model outperforms un- supervised baselines (including our own) by human evaluation along multiple attributes. 1 Introduction Recent approaches based on neural networks have brought significant advancements for both extrac- tive and abstractive summarization (Rush et al., 2015; Nallapati et al., 2016). However, their suc- cess relies on large-scale parallel corpora of input text and output summaries for direct supervision. For example, there are ~280,000 training instances Autoencoder Information Bottleneck Hong Kong, a bustling metropolis with a population over 7 million, was once under British Rule. Hong Kong has population over 7 million, was once under British Rule. Hong Kong was once under British Rule. Summary predict loss predict loss Source Summary Next The city returned to Chinese control in 1997. sentence Figure 1: Example contrasting the Autoencoder (AE) and Information Bottleneck (IB) approaches to summa- rization. While AE (top) preserves any detail that helps to reconstruct the original, such as population size in this example, IB (bottom) uses context to determine which information is relevant, which results in a more appropriate summary. in the CNN/Daily Mail dataset (Nallapati et al., 2016; Hermann et al., 2015), and ~4,000,000 in- stances in the sentence summarization dataset of Rush et al. (2015). Because it is too costly to have humans write gold summaries at this scale, existing large-scale datasets are based on naturally occurring pairs of summary-like text paired with source text, for instance using news titles or high- lights …

Content truncated to 500 words