HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information

Findings of the Association for Computational Linguistics: ACL 2022, pages 1292 - 1308 May 22-27, 2022 c⃝2022 Association for Computational Linguistics HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information Qian Ruan DFKI GmbH, Germany klararuan@gmail.com Malte Ostendorff DFKI GmbH, Germany malte.ostendorff@dfki.de Georg Rehm DFKI GmbH, Germany georg.rehm@dfki.de Abstract Transformer-based language models usually treat texts as linear sequences. However, most texts also have an inherent hierarchical struc- ture, i. e., parts of a text can be identified us- ing their position in this hierarchy. In addi- tion, section titles usually indicate the common topic of their respective sentences. We pro- pose a novel approach to formulate, extract, encode and inject hierarchical structure infor- mation explicitly into an extractive summariza- tion model based on a pre-trained, encoder- only Transformer language model (HiStruct+ model), which improves SOTA ROUGEs for extractive summarization on PubMed and arXiv substantially. Using various experimental set- tings on three datasets (i. e., CNN/DailyMail, PubMed and arXiv), our HiStruct+ model out- performs a strong baseline collectively, which differs from our model only in that the hier- archical structure information is not injected. It is also observed that the more conspicuous hierarchical structure the dataset has, the larger improvements our method gains. The ablation study demonstrates that the hierarchical posi- tion information is the main contributor to our model’s SOTA performance. 1 Introduction Texts, especially long documents, contain internal hierarchical structure like sections, paragraphs, sen- tences, and tokens. When we manually summarize a text, the hierarchical text structure usually plays a key role. Taking a scientific paper as an exam- ple, we might focus more on the sections with the titles of “methodology”, “discussion”, and “conclu- sion” while paying less attention to the sections like “background”. Furthermore, the sentences within one section could have closer relationship with each other, than the ones outside this section. Understanding not only the sequential relations between the sentences but also the internal hierar- chical text structure helps us better determine the important sentences within a document. Similarly, a neural summarization model could benefit from these hierarchical structure information. In this paper, we focus on extractive text sum- marization of single documents, which is the task of binary sentence classification with labels indi- cating whether a sentence should be included in a summary. Recently, pre-trained language mod- els based on Transformer (Vaswani et al., 2017), such as BERT (Devlin et al., 2019), have been widely used to extract contextual representations from texts. The pre-trained Transformer language models (TLMs) can be easily reused for fine- tuning on the downstream tasks, so that the rep- resentations already learned from the large pre- training corpora are preserved. Liu and Lapata (2019) have achieved the state-of-the-art (SOTA) performance by fine-tuning BERT for extractive summarization on short document datasets includ- ing CNN/DailyMail. However, the TLMs con- sider merely the sequential-context-dependency by adding a linear positional encoding to each input token embeddings. The hierarchical text structure information is not taken into account explicitly. We propose a novel approach to formulate, ex- …

Content truncated to 500 words