Extractive Text Summarization with Latent Topics using Heterogeneous Graph Neural Network

Extractive Text Summarization with Latent Topics using Heterogeneous Graph Neural Network Tuan-Anh Phan, Ngoc-Dung Nguyen, and Khac-Hoai Nam Bui ∗ Viettel Cyperspace Center, Viettel Group Hanoi, Vietnam {anhpt161,dungnn7,nambkh}@viettel.com.vn Abstract This paper presents a heterogeneous graph neural network (HeterGNN) model for extrac- tive text summarization (ETS) by using latent topics to capture the important content of in- put documents. Specifically, topical informa- tion has been widely used as global informa- tion for sentence selection. However, most of the recent approaches use neural models, which lead the training models more complex and difficult for extensibility. In this regard, this study presents a novel graph-based ETS by adding a new node of latent topics into Het- erGNN for summarization (TopicHeterGraph- Sum). Specifically, TopicHeterGraphSum in- cludes three types of semantic nodes (i.e., topic-word-sentence) in order to enrich the cross-sentence relations for extractive summa- rization. Furthermore, an extended version of TopicHeterGraphSum for multi documents extraction is also taken into account to empha- size the advantage of the proposed method. Experiments on benchmark datasets such as CNN/DailyMail and Multi-News show the promising results of our method compared with state-of-the-art models. 1 Introduction ETS is an important task of Natural Language Pro- cessing (NLP) in terms of extracting several relevant sentences from the original documents while keep- ing the main information. The traditional methods for ETS are TextRank (Mihalcea and Tarau, 2004) ∗Corresponding Author and LexRank (Erkan and Radev, 2004), which fo- cus on calculating the similarity between sentence scores. Sequentially, the rapid development of Deep Learning (DL) has brought breakthrough records by modeling a document as a sequence of sequences in order to deal with long-range inter-sentence re- lationships for the summarization (Cheng and La- pata, 2016; Cohan et al., 2018). However, cross- sentence relation is still a challenge in this research field (Liu and Lapata, 2019). Recent works focus on Graph Neural Networks (GNNs) (e.g., Graph Con- volutional Network (GCN) (Kipf and Welling, 2017) or Graph Attention Network (GAT) (Velickovic et al., 2018)) to explore the cross-sentence relation- ships for the summarization task. The core idea is to represent inter-sentential graphs and use message passing to extract the complex relationship in the in- put documents. For instance, (Yasunaga et al., 2017) and (Xu et al., 2020) adopt discourse analysis to build document graphs. (Jia et al., 2020) and (Wang et al., 2020) built a bi-partite graph between words and sentences, which is regarded as a heterogeneous graph neural network. Moreover, modeling global information is also taken into account for sentence selection by using pre-trained models (Liu and Lap- ata, 2019; Zhang et al., 2019). Sequentially, (Cui et al., 2020) utilized pre- trained BERT to learn contextual sentence represen- tations and train jointly with latent topics using the neural topic model (NTM). (Nguyen et al., 2021) presents an extended version using NTM for ab- stractive text summarization indicating the capabil- ity of enriching the global information for the sum- marization. Although the existing methods have provided re- markable results, there are several open research is- sues …

Content truncated to 500 words