Predicting Discourse Trees from Transformer-based Neural Summarizers

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4139–4152 June 6–11, 2021. ©2021 Association for Computational Linguistics 4139 Predicting Discourse Trees from Transformer-based Neural Summarizers Wen Xiao, Patrick Huber, Giuseppe Carenini Department of Computer Science University of British Columbia Vancouver, BC, Canada, V6T 1Z4 {xiaowen3, huberpat, carenini}@cs.ubc.ca Abstract Previous work indicates that discourse infor- mation benefits summarization. In this paper, we explore whether this synergy between dis- course and summarization is bidirectional, by inferring document-level discourse trees from pre-trained neural summarizers. In particu- lar, we generate unlabeled RST-style discourse trees from the self-attention matrices of the transformer model. Experiments across mod- els and datasets reveal that the summarizer learns both, dependency- and constituency- style discourse information, which is typically encoded in a single head, covering long- and short-distance discourse dependencies. Over- all, the experimental results suggest that the learned discourse information is general and transferable inter-domain1. 1 Introduction Extractive summarization is a common and im- portant task within the area of Natural Language Processing (NLP) , which can be useful in a multi- tude of diverse real-life scenarios. Current extrac- tive summarizers typically use exclusively neural approaches, in which the importance of extracted units (i.e., sentences or clauses) and relationship be- tween them are learned by the model from a large amount of data (e.g., Liu and Lapata (2019b)). Inspired by previous work in pre-neural times, indicating that discourse information, especially discourse trees according to the Rhetorical Struc- ture Theory (RST) (Mann and Thompson, 1988), can benefit the summarization task (Marcu, 1999), several very recent neural summarizers have tried to explicitly encode discourse information to sup- port summarization. Overall, it seems that adding these encodings, consistent with pre-neural results, is beneficial. In particular, injecting discourse has been shown to either improve performance 1The code can be found in https://github.com/ Wendy-Xiao/summ_guided_disco_parser on the extractive summarization task itself (Xu et al., 2020), or allow for a substantial reduction in the number of the summarizer’s parameters, while keeping competitive performance (Xiao et al., 2020). The central hypothesis we are exploring in this paper is whether the synergy between discourse parsing and summarization is bidirectional. In other words, we examine if summarization is a useful auxiliary task to infer discourse structures. Liu et al. (2019b) performed a preliminary inves- tigation of this conjecture, showing that structural information can be inferred from attention mech- anisms while training a neural model on auxiliary tasks. However, they did not perform any compari- son against ground-truth discourse trees. Further, recent work showed that discourse trees implicitly induced during training are oftentimes trivial and shallow, not representing valid discourse structures (Ferracane et al., 2019). In this paper, we address these limitations by explicitly exploring the relationship between sum- marization and discourse parsing through the infer- ence of document-level discourse trees from pre- trained summarization models, comparing the re- sults against ground-truth RST discourse trees. Be- sides Liu et al. (2019b), our idea and approach are inspired by …

Content truncated to 500 words