Modeling Content Importance for Summarization with Pre-trained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3606–3611, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3606 Modeling Content Importance for Summarization with Pre-trained Language Models Liqiang Xiao1, Lu Wang2, Hao He1,3∗, Yaohui Jin1,3∗ 1MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2Computer Science and Engineering, University of Michigan 3State Key Lab of Advanced Optical Communication System and Network, Shanghai Jiao Tong University xiaoliqiang@sjtu.edu.cn, wangluxy@umich.edu {hehao, jinyh}@sjtu.edu.cn Abstract Modeling content importance is an essential yet challenging task for summarization. Previ- ous work is mostly based on statistical meth- ods that estimate word-level salience, which does not consider semantics and larger con- text when quantifying importance. It is thus hard for these methods to generalize to seman- tic units of longer text spans. In this work, we apply information theory on top of pre- trained language models and define the con- cept of importance from the perspective of in- formation amount. It considers both the se- mantics and context when evaluating the im- portance of each semantic unit. With the help of pre-trained language models, it can eas- ily generalize to different kinds of semantic units (n-grams or sentences). Experiments on CNN/Daily Mail and New York Times datasets demonstrate that our method can bet- ter model the importance of content than prior work based on F1 and ROUGE scores. 1 Introduction and Related Work Text summarization aims to compress long docu- ment(s) into a concise summary while maintaining the salient information. It often consists of two crit- ical subtasks, important information identification and natural language generation (for abstractive summarization). With the advancements of large pre-trained language models (PreTLMs) (Devlin et al., 2019; Yang et al., 2019), state-of-the-art re- sults are achieved on both natural language under- standing and generation. However, it is still unclear how well these large models can estimate “content importance” for a given document. Previous studies for modeling importance are either empirical-based, which implicitly encode importance during document summarization, or theory-based, which often lacks support by empiri- cal experiments (Peyrard, 2019). Benefiting from ∗Corresponding author the large-scale summarization datasets (Nallapati et al., 2016; Narayan et al., 2018), data-driven ap- proaches (Nallapati et al., 2017; Paulus et al., 2018; Zhang et al., 2019) have made significant progress. Yet most of them conduct the information selec- tion implicitly while generating the summaries. It lacks theory support and is hard to be applied to low-resource domains. In another line of work, structure features (Zheng and Lapata, 2019), such as centrality, position, and title, are employed as proxies for importance. However, the information captured by these features can vary in texts of dif- ferent genres. To overcome this problem, theory-based meth- ods (Louis, 2014; Peyrard, 2019; Lin et al., 2006) aim to formalize the concept of importance, and develop general-purpose systems by modeling the background knowledge of readers. This is based on the intuition that humans are good at identifying important content by using their own interpreta- tion of the world knowledge. …

Content truncated to 500 words