Abstractive Summarization Guided by Latent Hierarchical Document Structure

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5303 - 5317 December 7-11, 2022 ©2022 Association for Computational Linguistics Abstractive Summarization Guided by Latent Hierarchical Document Structure Yifu Qiu Shay B. Cohen Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB Y.QIU-20@sms.ed.ac.uk, scohen@inf.ed.ac.uk Abstract Sequential abstractive neural summarizers of- ten do not use the underlying structure in the input article or dependencies between the input sentences. This structure is essential to inte- grate and consolidate information from differ- ent parts of the text. To address this shortcom- ing, we propose a hierarchy-aware graph neural network (HierGNN) which captures such de- pendencies through three main steps: 1) learn- ing a hierarchical document structure through a latent structure tree learned by a sparse matrix- tree computation; 2) propagating sentence in- formation over this structure using a novel message-passing node propagation mechanism to identify salient information; 3) using graph- level attention to concentrate the decoder on salient information. Experiments confirm Hi- erGNN improves strong sequence models such as BART, with a 0.55 and 0.75 margin in aver- age ROUGE-1/2/L for CNN/DM and XSum. Further human evaluation demonstrates that summaries produced by our model are more relevant and less redundant than the baselines, into which HierGNN is incorporated. We also find HierGNN synthesizes summaries by fus- ing multiple source sentences more, rather than compressing a single source sentence, and that it processes long inputs more effectively.1 1 Introduction Sequential neural network architectures in their various forms have become the mainstay in abstrac- tive summarization (See et al., 2017; Lewis et al., 2020). However, the quality of machine-produced summaries still lags far behind the quality of human summaries (Huang et al., 2020a; Xie et al., 2021; Cao et al., 2022; Lebanoff et al., 2019). Due to their sequential nature, a challenge with neural summa- rizers is to capture hierarchical and inter-sentential dependencies in the summmarized document. 1Code is available at https://github.com/yfqiu-nlp/hiergnn. Article Sentences: 1. The town is home to the prestigious Leander Club, which has trained more than 100 Olympic medal-winning rowers. - 2 sentences are abbreviated here. 4. The Royal Mail has painted more than 50 postboxes gold following Team GB’s gold medal haul at London 2012. 5. Originally it said it was only painting them in winners home towns, or towns with which they are closely associated. 6. Town mayor Elizabeth Hodgkin said: “ We are the home of rowing ... I feel very excited about it." - 5 sentences are abbreviated here. 12. The Henley-on-Thames postbox was painted on Friday. - one sentence is abbreviated here. Reference Summary: The Royal Mail has painted a postbox gold in the Oxford-shire town of Henley-on-Thames - in recognition of its medal winning rowing club. BART’s Summary: A postbox in Henley-on-Thames has been painted gold as part of the Royal Mail’s “ Olympic gold ” cam- paign. Our HierGNN’s Summary: A Royal Mail postbox in Henley-on- Thames has been painted gold in …

Content truncated to 500 words