GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization

Proceedings of the 29th International Conference on Computational Linguistics, pages 6259–6269 October 12–17, 2022. 6259 GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization Qianqian Xie National Centre for Text Mining, Department of Computer Science, The University of Manchester qianqian.xie@manchester.ac.uk Jimin Huang Chancefocus AMC. jimin@chancefocus.com Tulika Saha and Sophia Ananiadou National Centre for Text Mining, Department of Computer Science, The University of Manchester {tulika.saha,sophia.ananiadou}@manchester.ac.uk Abstract Recently, neural topic models (NTMs) have been incorporated into pre-trained language models (PLMs), to capture the global seman- tic information for text summarization. How- ever, in these methods, there remain limitations in how they capture and integrate the global semantic information. In this paper, we pro- pose a novel model, Graph contRastivE Topic Enhanced Language model (GRETEL), that incorporates the graph contrastive topic model with the pre-trained language model, to fully leverage both the global and local contextual semantics for long document extractive sum- marization. To better capture and incorporate the global semantic information into PLMs, the graph contrastive topic model integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the seman- tic information from the global document con- text and the gold summary. To this end, GRE- TEL encourages the model to efficiently extract salient sentences that are topically related to the gold summary, rather than redundant sentences that cover sub-optimal topics. Experimental re- sults1 on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods. 1 Introduction Due to the well-known limitation of pre-trained lan- guage models (PLMs) (Devlin et al., 2019; Wang et al., 2021) that they fail to capture long-range de- pendencies (Beltagy et al., 2020), attempts have been proposed to integrate neural topic models 1https://github.com/xashely/GRETEL_ extractive (NTMs) (Cao et al., 2015; Peng et al., 2018; Xie et al., 2021) into PLMs, which have shown signif- icant improvement in the performance of the text summarization task (Wang et al., 2020b; Cui and Hu, 2021b; Nguyen et al., 2021; Fu et al., 2020). In addition to the local contextual information cap- tured in PLMs, NTMs can provide an approxima- tion of the global semantics captured from docu- ment contents, i.e., latent topics, as well as their posterior topic representations. The global seman- tics are further used to guide the model, to generate coherent summaries which cover the most relevant topics discussed within the document, via the at- tention mechanism (Wang et al., 2020b; Aralikatte et al., 2021; Nguyen et al., 2021; Fu et al., 2020) or graph neural networks (GNNs) (Cui and Hu, 2021b; Cui et al., 2020). However, there exists the semantic gap between latent topics as the approximation global seman- tics, and the true global semantics due to two major limitations for existing methods. The first limita- tion concerns the nature of unsupervised topic inference in these methods, where topics and pos- terior topic distributions are learned from docu- ments in an unsupervised manner, without consid- ering the key semantic information conveyed in the gold summary (mostly the abstract). Existing …

Content truncated to 500 words