Unsupervised Extractive Summarization using Pointwise Mutual Information

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2505–2512 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2505 Unsupervised Extractive Summarization using Pointwise Mutual Information Vishakh Padmakumar New York University vishakh@nyu.edu He He New York University hehe@cs.nyu.edu Abstract Unsupervised approaches to extractive summa- rization usually rely on a notion of sentence importance defined by the semantic similarity between a sentence and the document. We pro- pose new metrics of relevance and redundancy using pointwise mutual information (PMI) be- tween sentences, which can be easily com- puted by a pre-trained language model. Intu- itively, a relevant sentence allows readers to infer the document content (high PMI with the document), and a redundant sentence can be inferred from the summary (high PMI with the summary). We then develop a greedy sentence selection algorithm to maximize rel- evance and minimize redundancy of extracted sentences. We show that our method outper- forms similarity-based methods on datasets in a range of domains including news, medical journal articles, and personal anecdotes. 1 Introduction Modern neural network-based approaches to sum- marization require a large amount of document- summary pairs that are usually unavailable outside of the news domain. For example, summariza- tion datasets of personal narratives and office meet- ings contain only a few hundred examples (Ouyang et al., 2017; Carletta et al., 2005). In this work, we tackle the problem of unsupervised extractive summarization which aims to select important sen- tences from the document. While there exists ex- tensive prior work (Radev et al., 2000; Mihalcea and Tarau, 2004; Liu and Lapata, 2019; Zheng and Lapata, 2019), most approaches rely on the assump- tion that important sentences are similar to other sentences in the document. However, it is unclear if similarity-based features lead to meaningful con- tent selection (Kedzie et al., 2018). Inspired by recent work on formalizing the no- tion of importance in summarization (Peyrard, 2019), we propose metrics for relevance and re- dundancy based on pointwise mutual informa- tion (PMI). Intuitively, a relevant summary allows the reader to maximally infer the document con- tent, and a summary has minimal redundancy if each sentence in it provides additional informa- tion. Therefore, we measure the relevance of a summary by its PMI with the document. High rele- vance means that the probability of the document increases conditioning on the summary. Similarly, we measure redundancy by PMI of sentence pairs within the summary. A sentence is redundant if seeing other sentences significantly increases its probability. Based on the new metrics, we design a sim- ple sentence extraction algorithm. We estimate the PMI of sentence pairs by a pre-trained lan- guage model fine-tuned on in-domain documents. We then use a simple sequential sentence selec- tion algorithm for extractive summarization, which greedily maximizes relevance and minimizes re- dundancy. Experimental results show that our algorithm outperforms similarity-based methods across multi- ple domains, including news, personal stories, and medical articles.1 2 Relevance and Redundancy We begin by formalizing relevance and redundancy …

Content truncated to 500 words