HEGEL: Hypergraph Transformer for Long Document Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10167 - 10176 December 7-11, 2022 ©2022 Association for Computational Linguistics HEGEL: Hypergraph Transformer for Long Document Summarization Haopeng Zhang, Xiao Liu, Jiawei Zhang IFM Lab, Department of Computer Science, University of California, Davis, CA, USA haopeng,xiao,jiawei@ifmlab.org Abstract Extractive summarization for long documents is challenging due to the extended structured input context. The long-distance sentence de- pendency hinders cross-sentence relations mod- eling, the critical step of extractive summa- rization. This paper proposes HEGEL, a hy- pergraph neural network for long document summarization by capturing high-order cross- sentence relations. HEGEL updates and learns effective sentence representations with hyper- graph transformer layers and fuses different types of sentence dependencies, including la- tent topics, keywords coreference, and section structure. We validate HEGEL by conduct- ing extensive experiments on two benchmark datasets, and experimental results demonstrate the effectiveness and efficiency of HEGEL. 1 Introduction Extractive summarization aims to generate a shorter version of a document while preserving the most salient information by directly extracting rel- evant sentences from the original document. With recent advances in neural networks and large pre- trained language models (Devlin et al., 2018; Lewis et al., 2019), researchers have achieved promis- ing results in news summarization (around 650 words/document) (Nallapati et al., 2016a; Cheng and Lapata, 2016; See et al., 2017; Zhang et al., 2022; Narayan et al., 2018; Liu and Lapata, 2019). However, these models struggle when applied to long documents like scientific papers. The input length of a scientific paper can range from 2000 to 7, 000 words, and the expected summary (abstract) is more than 200 words compared to 40 words in news headlines. Scientific paper extractive summarization is highly challenging due to the long structured in- put. The extended context hinders sequential mod- els like RNN from capturing sentence-level long- distance dependency and cross-sentence relations, Figure 1: An illustration of modeling cross-sentence re- lations from section structure, latent topic, and keyword coreference perspectives. which are essential for extractive summarization. In addition, the quadratic computation complexity of attention with respect to the input tokens length makes Transformer (Vaswani et al., 2017) based models not applicable. Moreover, long documents typically cover diverse topics and have richer struc- tural information than short news, which is difficult for sequential models to capture. As a result, researchers have turned to graph neural network (GNN) approaches to model cross- sentence relations. They generally represent a doc- ument with a sentence-level graph and turn extrac- tive summarization into a node classification prob- lem. These work construct graph from document in different manners, such as inter-sentence cosine similarity graph in (Erkan and Radev, 2004; Dong et al., 2020), Rhetorical Structure Theory (RST) tree relation graph in (Xu et al., 2019), approximate discourse graph in (Yasunaga et al., 2017), topic- sentence graph in (Cui and Hu, 2021) and word- document heterogeneous graph in (Wang et al., 2020). However, the usability of these approaches 10167 is limited by the following two aspects: (1) …

Content truncated to 500 words