Multi Graph Neural Network for Extractive Long Document Summarization

Proceedings of the 29th International Conference on Computational Linguistics, pages 5870–5875 October 12–17, 2022. 5870 Multi Graph Neural Network for Extractive Long Document Summarization Xuan-Dung Doan1 and Le-Minh Nguyen2 and Khac-Hoai Nam Bui1∗ 1Viettel Cyperspace Center, Viettel Group, Vietnam 2Japan Advanced Institute of Science and Technology, Japan dungdx4@viettel.com.vn,nguyenml@jaist.ac.jp,nambkh@viettel.com.vn Abstract Heterogeneous Graph Neural Networks (Het- erGNN) have been recently introduced as an emergent approach for extracting document summarization (EDS) by exploiting the cross- relations between words and sentences. How- ever, applying HeterGNN for long documents is still an open research issue. One of the main majors is the lacking of inter-sentence connec- tions. In this regard, this paper exploits how to apply HeterGNN for long documents by build- ing a graph on sentence-level nodes (homoge- neous graph) and combine with HeterGNN for capturing the semantic information in terms of both inter and intra-sentence connections. Ex- periments on two benchmark datasets of long documents such as PubMed and ArXiv show that our method is able to achieve state-of-the- art results in this research field. 1 Introduction Extractive Document summarization aims to auto- matically extract a set of sentences, which repre- sents information for the whole document, by rank- ing the importance of sentence features. Recent works focus on GNN, a Deep learning-based ap- proach that operates on graph domain (Zhou et al., 2020), to achieve remarkable results in this research field. Specifically, GNN-based models are able to encode the complicated pairwise relationships be- tween entity tokens for better informative represen- tations (Wu et al., 2021). Cui et al. (2020) uses information of topic-aware to change the represen- tation of words to a new representation. Then, a GNN-based model is presented for capturing re- lationships efficiently via graph-structured docu- ment representation between sentences. Sequen- tially, HeterGNN, a special kind of GNN (Zhang et al., 2019), has been proposed as a promising ap- proach to enrich the relationships between words and sentences. Wang et al. (2020) introduced HSG, ∗Corresponding author a heterogeneous graph-based neural network for ex- tractive summarization by using more fine-grained semantic units in the summarization graph to ex- tract the complex relationships between words and sentences. Accordingly, the model has achieved the top performance in CNN/DailyMail and NYT50 datasets in terms of the non-BERT-based approach. In order to utilize the capability of BERT-based models (Devlin et al., 2019), Jia et al. (2020) pro- posed a hierarchical attentive heterogeneous graph (HAHSum) to improve the redundant phrases prob- lem between extracted sentences of the summariza- tion, which has achieved promising results on news article datasets such as CNN/DailyMail, NYT, and Newsroom. Nevertheless, the model requires exter- nal analysis for modeling long-range dependencies. Intuitively, transformer-based language models are not able to process long pieces of text. Several works have provided promising results (Cui and Hu, 2021), however, the input length limitation and encoding of long texts are still open challenges in this research field (Zhong et al., 2020). In this study, we take an investigation on im- proving the performance of the EDS problem …

Content truncated to 500 words