Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5095–5112 July 9-14, 2023 ©2023 Association for Computational Linguistics Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization Pengcheng He1, Baolin Peng2, Song Wang1, Yang Liu1, Ruochen Xu1, Hany Hassan Awadalla1,Yu Shi1, Chenguang Zhu1, Wayne Xiong1, Michael Zeng1, Jianfeng Gao2, Xuedong Huang1 1 Microsoft Azure AI 2 Microsoft Research penhe@microsoft.com Abstract This paper presents Z-Code++, a new pre- trained language model optimized for abstrac- tive text summarization. The model extends the state of the art encoder-decoder model us- ing three techniques. First, we use a two-phase pre-training process to improve model’s perfor- mance on low-resource summarization tasks. The model is first pre-trained using text cor- pora for language understanding, and then is continually pre-trained on summarization cor- pora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierar- chical manner. Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. Our model is parameter- efficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models. 1 Introduction Text summarization aims at producing a concise and fluent summary while preserving salient con- tent and overall meaning of the source docu- ments. It has been applied in a wide range of real-world applications, e.g., summarizing Web search results for interactive information retrieval (Gao et al., 2022) and generating medical sum- maries from doctor-patient conversation transcripts (Zhang et al., 2021). While the extractive approach is the dominant approach in commercial systems due to its simplic- ity and effectiveness (Allahyari et al., 2017), the abstractive approach is getting more attention in the research community as neural language models are used (e.g., Rush et al., 2015; Nallapati et al., 2016; Chopra et al., 2016; Liu and Lapata, 2019b,a; Pasunuru et al., 2021). Compared to the extractive approach where a summary is constructed using extracted sentences, abstractive summarizers para- phrase the idea of the source documents in a new form, and have a potential of generating more con- cise and coherent summaries. However, good abstractive summarizers are harder to develop since we have to deal with prob- lems like semantic representation, inference and low-resource text generation, which are more chal- lenging than sentence extraction. Recently, large- scale pre-trained language models (PLMs) such as PEGASUS (Zhang et al., 2020), GPT (Radford et al., 2019; Brown et al., 2020), T5 (Raffel et al., 2020), have been applied for abstractive summa- rization. While these models can produce surpris- ingly fluent text, the generated summaries often contain factual inconsistencies, caused by distorted or fabricated facts about the source documents, which is known as the hallucination problem (Kry´s- ci´nski …

Content truncated to 500 words