Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT

Proceedings of the 28th International Conference on Computational Linguistics, pages 5629–5639 Barcelona, Spain (Online), December 8-13, 2020 5629 Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT Ruifeng Yuan Zili Wang Wenjie Li Department of Computing, The Hong Kong Polytechnic University, Hong Kong Xidian University {csryuan, cswjli}@comp.polyu.edu.hk ziliwang.do@gmail.com Abstract Most current extractive summarization models generate summaries by selecting salient sen- tences. However, one of the problems with sentence-level extractive summarization is that there exists a gap between the human-written gold summary and the oracle sentence labels. In this paper, we propose to extract fact-level semantic units for better extractive summarization. We also introduce a hierarchical structure, which incorporates the multi-level of granularities of the textual information into the model. In addition, we incorporate our model with BERT using a hierarchical graph mask. This allows us to combine BERT’s ability in natural language under- standing and the structural information without increasing the scale of the model. Experiments on the CNN/DaliyMail dataset show that our model achieves state-of-the-art results. 1 Introduction The aim of text summarization is to generate a compressed shorter highlight of a given document. The generated summaries must conform to natural language constraints and cover the most important infor- mation conveyed in the source text. In this paper, we focus on extractive summarization which generates summaries by selecting salient sentences or other semantic units from the source text. As we know, sentences are regarded as the basic textual units making up the final summaries in most extractive summarization models. One of the problems with sentence-level extractive summarization is that there exists a gap between the human-written gold summary and the training objective. Since most datasets do not contain sentence-level labels, oracle summaries are normally generated by a greedy algorithm, which maximizes the ROUGE score between the sentences in the source text and in the gold summary (Nallapati et al., 2017; Liu and Lapata, 2019). As pointed out by (Narayan et al., 2018), sentences with high ROUGE scores may not necessarily lead to an optimal summary. Such discrepancy may be due to the overlapping contents and over extraction. Therefore, some researches try to perform extraction at a lower level such as words or phrases (Bui et al., 2016; Cheng and Lapata, 2016; Gehrmann et al., 2018). Although these models are able to learn which phrases or words are more important directly from the gold summary, it is hard to maintain semantic integrity when information is scattered. Inspired by (Cao et al., 2018), we propose to extract facts, a granularity between phrases and sen- tences, as the primary textual units to generate summaries. In this work, we first develop a heuristic algorithm to split a sentence into several smaller units, where each unit is considered as a single fact description. Then we apply a one-to-one strategy to match each fact in the gold summary to one fact in the source text to obtain the oracle label. In this way, we smooth the gap between the gold summary and oracle labels and …

Content truncated to 500 words