Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4039–4044 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4039 Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer Jingun Kwon, Naoki Kobayashi, Hidetaka Kamigaito and Manabu Okumura Tokyo Institute of Technology {kwonjingun,kobayasi,kamigaito}@lr.pi.titech.ac.jp oku@pi.titech.ac.jp Abstract Sentence extractive summarization shortens a document by selecting sentences for a sum- mary while preserving its important contents. However, constructing a coherent and informa- tive summary is difficult using a pre-trained BERT-based encoder since it is not explicitly trained for representing the information of sen- tences in a document. We propose a nested tree-based extractive summarization model on RoBERTa (NeRoBERTa), where nested tree structures consist of syntactic and discourse trees in a given document. Experimental re- sults on the CNN/DailyMail dataset showed that NeRoBERTa outperforms baseline mod- els in ROUGE. Human evaluation results also showed that NeRoBERTa achieves signifi- cantly better scores than the baselines in terms of coherence and yields comparable scores to the state-of-the-art models. 1 Introduction Document summarization is a task of creating a concise summary from a given document while keeping the original content. In general, sentence extraction methods, which select sentences in a doc- ument to create its summary, have the advantages of truthfulness compared with abstractive methods (Cao et al., 2018) and of fluency compared with word extraction methods (Xu et al., 2020). Neural networks have achieved great success in sentence extraction-based document summariza- tion (Cheng and Lapata, 2016; Zhou et al., 2018). Recently, Liu and Lapata (2019) proposed BERT- SUM, which utilizes BERT (Devlin et al., 2019) for sentence representations to create a summary. Although the use of BERT resulted in significant performance improvement, this method decides the selection for each sentence independently. Xu et al. (2020) proposed DISCOBERT by consider- ing inter-sentence information through discourse graphs to construct a coherent summary. Although they achieved remarkable scores in ROUGE, it was ⋯ 𝑤!" 𝑤!!⋯ 𝑤!# ⋯ ⋯𝑤#" 𝑤#! 𝑤## 𝑟𝑜𝑜𝑡# ⋯ 𝑤$!⋯𝑤$" 𝑤$# 𝑆# 𝑟𝑜𝑜𝑡! 𝑟𝑜𝑜𝑡$ ⋯ ⋯ 0 1 0 ⋯ Intra 𝑆! 𝑆$ (DiscoBERT) Inter (NeRoBERTa) Nested (1: select, 0: delete) Figure 1: Different from the previous work, DIS- COBERT (Xu et al., 2020), NeRoBERTa selects sen- tences by considering both intra- and inter-sentence re- lationships as a nested tree structure. still difficult to construct a coherent summary com- pared to BERTSUM in human evaluation. Zhong et al. (2020) attempted to change the paradigm by formulating summary-level extraction with a RoBERTa encoder and achieved the state-of-the- art results on the CNN/DailyMail dataset. In spite of the successful results of the above BERT-related methods, their sentence representa- tions have room for improvement. As Liu et al. (2019) reported, “[CLS]”, a pre-defined token for indicating sentence representations on BERT, is insufficient to express sentence information. Even in RoBERTa, it is also a problem due to the lack of next sentence prediction in its pretraining step. Therefore, for further improving summarization performance, we need to consider how to repre- sent sentences in a BERT-related …

Content truncated to 500 words