HeterGraphLongSum: Heterogeneous Graph Neural Network with Passage Aggregation for Extractive Long Document Summarization

Proceedings of the 29th International Conference on Computational Linguistics, pages 6248–6258 October 12–17, 2022. 6248 HeterGraphLongSum: Heterogeneous Graph Neural Network with Passage Aggregation for Extractive Long Document Summarization Tuan-Anh Phan and Ngoc-Dung Nguyen and Khac-Hoai Nam Bui∗ Viettel Cyperspace Center, Viettel Group, Vietnam {anhpt161,dungnn7,nambkh}@viettel.com.vn Abstract Graph Neural Network (GNN)-based models have proven effective in various Natural Lan- guage Processing (NLP) tasks in recent years. Specifically, in the case of the Extractive Doc- ument Summarization (EDS) task, modeling documents under graph structure is able to an- alyze the complex relations between seman- tic units (e.g., word-to-word, word-to-sentence, sentence-to-sentence) and enrich valuable infor- mation for the sentence representation. How- ever, long-form document summarization us- ing graph-based approaches is still an open re- search issue. The main challenge is to represent long documents in a graph structure in an effec- tive way. In this regard, this paper proposes a new heterogeneous graph neural network (Het- erGNN) model to improve the performance of long document summarization (HeterGraph- LongSum). Specifically, the main idea is to add the passage nodes into the heterogeneous graph structure of word and sentence nodes for enriching the final representation of sen- tences. In this regard, HeterGraphLongSum includes three types of semantic units such as word, sentence, and passage. Experiments on two benchmark datasets for long documents such as Pubmed and Arxiv indicate promising results of the proposed model for the extractive long document summarization problem. Espe- cially, HeterGraphLongSum is able to achieve state-of-the-art performance without relying on any pre-trained language models (e.g., BERT). The source code is available for further exploita- tion on the Github1. 1 Introduction Document summarization is one of the central problems in NLP, which aims to rewrite a single document or multi documents under a shorter ver- sion with preserving the main information. There are two major approaches such as extractive and ∗corresponding author 1https://github.com/tuananhphan97vn/HeterGraphLongSum abstractive summarization. Abstractive models are more sophisticated abilities that require well- comprehensive reading text and generating high- quality text. Specifically, most of the existing ar- chitectures have been built based on sequence-to- sequence (Seq2Seq) techniques in different ways such as Recurrent Neural Network (RNN) (Nalla- pati et al., 2017), Pointer-Generator-Network(See et al., 2017), or Transformer-based models(Zhang et al., 2020; Xiao and Carenini, 2020). Further- more, the external information, for instance, pre- trained model BERTSum(Liu and Lapata, 2019) and topic modeling (Wang et al., 2020b; Nguyen et al., 2021)) are incorporated to improve perfor- mances. Nevertheless, this approach requires a complicated neural network that consists of mil- lions of learnable parameters, which is the cause of raising significant costs in both terms of com- putation time perplexity and resources. Therefore, extractive models still gain much attention. Partic- ularly, extractive document summarization (EDS) takes a document in the form list of sentences and chooses several best candidates from the original document, then combine them to create the sum- marization. Recent models trend to turn EDS into the sequential binary-labeling task (Nallapati et al., 2017; Cheng and Lapata, 2016; Zhou et al., 2018). Graph …

Content truncated to 500 words