Incorporating Linguistic Knowledge for Abstractive Multi-document Summarization

Incorporating Linguistic Knowledge for Abstractive Multi-document Summarization Congbo Ma, Wei Emma Zhang, Hu Wang, Shubham Gupta and Mingyu Guo The University of Adelaide, Adelaide, Australia {congbo.ma,wei.e.zhang,hu.wang,a1787223,mingyu.guo}@adelaide.edu.au Abstract Linguistic knowledge plays an important role in assisting models to learn informative rep- resentations that could help guide better nat- ural language generation. In this paper, we develop a Transformer-based abstractive multi-document summarization method with linguistic-guided attention (LGA) mechanism for better representation learning. The pro- posed linguistic-guided attention mechanism can be seamlessly incorporated into multi- ple mainstream Transformer-based summa- rization models to improve the quality of the generated summaries. We develop the pro- posed method based on Flat Transformer (FT) and Hierarchical Transformer (HT), named ParsingSum-FT and ParsingSum-HT respec- tively. Empirical studies on both models demonstrate this simple but effective mecha- nism can help the models outperform existing Transformer-based methods on the benchmark datasets by a large margin. Extensive analyses examine different settings and configurations of the proposed model, providing a good ref- erence to the text summarization community. 1 Introduction Multi-document summarization (MDS) is a critical task in natural language processing aiming at gener- ating a informative summary from a set of content- related documents. There are two types of summary generation: extractive summarization by selecting salient sentences from original texts directly and ab- stractive summarization to generate summaries by models from the understanding of the input con- tents (Ma et al., 2022). Under comparison, abstrac- tive summarization is more challenging because it requires models to truly understand the input docu- ments and generate corresponding summaries. With the development of deep learning techniques, neural network-based models that can help to capture high- quality latent features are widely applied in MDS (Dhakras et al., 2018; Alexander et al., 2019; Liu et al., 2019; Li et al., 2020; Han et al., 2020; Wen et al., 2021; Beltrachini et al., 2021). Recently, Transformer (Ashish et al., 2017) shows outstanding performances in various natural lan- guage processing tasks, and it is also introduced into MDS (Alexander et al., 2019). Transformer has nat- ural advantages for parallelization and could retain long-range relations between pairs of tokens among documents. Liu et al. (Liu et al., 2018) adopted a Transformer model to generate Wikipedia articles. The model selects top-K tokens and feeds them into the decoder-only sequence transduction. Built upon this work, Liu et al. (Liu et al., 2019) proposed a Hi- erarchical Transformer (HT) containing token-level and paragraph-level Transformer layers for cross- document relations capturing. Wen et al. (Wen et al., 2021) proposed a pre-train language model PRIMERA, using encoder-decoder transformers to simplify the processing of concatenated input doc- uments, leverages the Longformer (Beltagy et al., 2020) to pre-train with a novel entity-based sentence masking objective. However, computing token-wise self-attention in the Transformer takes pairs of to- ken relations into account but lacks syntactic support that may cause content irrelevance and deviation for summary generation (Jin et al., 2020). Table 1: Generated summaries via different MDS mod- els. Different colors mean different thought groups. Source Docu- ments a …

Content truncated to 500 words