Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5574–5590 July 9-14, 2023 ©2023 Association for Computational Linguistics Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization Dongqi Pu Yifan Wang Vera Demberg Department of Computer Science Department of Language Science and Technology Saarland Informatics Campus, Saarland University, Germany {dongqipu,yifwang,vera}@lst.uni-saarland.de Abstract For text summarization, the role of discourse structure is pivotal in discerning the core con- tent of a text. Regrettably, prior studies on in- corporating Rhetorical Structure Theory (RST) into transformer-based summarization models only consider the nuclearity annotation, thereby overlooking the variety of discourse relation types. This paper introduces the ‘RSTformer’, a novel summarization model that comprehen- sively incorporates both the types and uncer- tainty of rhetorical relations. Our RST-attention mechanism, rooted in document-level rhetor- ical structure, is an extension of the recently devised Longformer framework. Through rig- orous evaluation, the model proposed herein ex- hibits significant superiority over state-of-the- art models, as evidenced by its notable perfor- mance on several automatic metrics and human evaluation.1 1 Introduction For writing a good summary of a long docu- ment, it is of paramount importance to discern the salient information within the text and to com- prehend the intricate interconnections among its various components. Contemporary leading-edge systems for abstractive (long) text summarization employ Transformer (Vaswani et al., 2017) encoder- decoder architecture (Zaheer et al., 2020; Guo et al., 2022). These sequence-to-sequence (seq2seq) mod- els first transform the source document into a high- dimensional content representation and then de- code the predicted summary conditioned on the representation (Belinkov and Bisk, 2018; Xu and Durrett, 2019; Cao and Wang, 2022; Balachandran et al., 2021). It has been demonstrated in the past that such an architecture does a poor job of digging high-level discourse structure during the encoding phase (Lin et al., 2019; Zhang et al., 2020; Koto 1The project information can be accessed by visiting: https://dongqi.me/projects/RSTformer. Figure 1: An example of RST tree: [Rhetorical structure theory (RST) is a theory of text organization.]EDU1 [Al- though the RST structure is difficult to annotate,]EDU2 [there are still many scholars who have studied it.]EDU3 et al., 2021a; de Wynter et al., 2023). However, discourse structure is very important for deciding what to include vs. not to include in the summary (Marcu, 1997, 1999, 1998; Zhong et al., 2020). Given that previous work has indicated that the performance of neural language models can be en- hanced through the incorporation of latent structure information (Ettinger, 2020; Miaschi et al., 2020; Qian et al., 2021; Pu and Sima’an, 2022), we will here explore the integration of discourse relation structure into the Longformer model (Beltagy et al., 2020); this architecture has been shown to be par- ticularly suitable for encoding long input texts. Rhetorical Structure Theory (RST) serves as a discourse framework designed to articulate the in- terrelationships among sentences at the document level. This framework distinguishes a plethora of coherence relations delineating the manner in which two text segments are …

Content truncated to 500 words